Logstash monitoring warning

[gsk]

Hi all,

I have a cluster of three master-eligible nodes, which are also data nodes, and one coordinating node which also runs a kibana instance. There's also a pipeline node. It's about 7.8.0 version.

All the cluster components are sending monitoring data to the coordinating node over port 9200/tcp. I can see components health in the kibana monitoring page. However, for logstash monitoring I get the following warning:

[WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@my-mastereligible-data-node.mydomain.tld:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@my-mastereligible-data-node.mydomain.tld:9200/][Manticore::SocketException] No route to host (Host unreachable)"}

I get this warning for each master-eligible/data node. If I open the port 9200 the connections are established and the warnings are gone.

My question is why logstash needs to communicate over port 9200 with master-eligible/data nodes. I was expecting the communication with the coordinating node over port 9200 being enough. What makes it strange to me is the fact that (as mentioned above) I can see logstash health in kibana page, which means monitoring is working. Am I missing something?

[gsk]

Another question rising up after the previous one is how logstash knows about the master-eligible/data nodes in my cluster, since the only node configured for logstash to send data to is the coordinating node. To my understanding, logstash is not a cluster node, it doesn't join the cluster, instead it is just in charge of pipeline functionalities. Shouldn't logshtash know nothing about the other nodes in the cluster?