Install metricbeat agent on all ES nodes in my cluster

[bkasrai]

Good afternoon,
I have installed Metricbeat on an individual node within my ES cluster and was able to get the ES module enabled along with the metrics\dashboards sent to Kibana. I am looking to expand this into the cluster and work to get it installed on the remaining five nodes. I have been unable to succeed at this as I have not seen data and have it sent over to Kibana. I was doing some reading and noticed that a monitoring cluster must be enabled\configured to have data sent from the PROD cluster to the Monitoring cluster however we do not have that capability. I also was curious about the output.elasticsearch setting for each node, is this where I put each ES node or can I set a single node for all nodes to connect to?

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:

Array of hosts to connect to.

hosts: ["localhost:9200"]

Optional protocol and basic auth credentials.

#protocol: "https"
#username: "elastic"
#password: "changeme"

Please let me know what other information you need from me so I can further assist. I am just looking to get data from each node reported over into Kibana.

[shaunak (Shaunak Kashyap)]

Hi,

Quick clarification question to make sure I take this in the right direction: when you refer to dashboards in Kibana, are you referring to the Stack Monitoring UI that ships with Kibana (Basic license and above) OR are you referring to custom dashboards that you've created?

Thanks,

Shaunak

[bkasrai]

My apologies for not clarifying that earlier. This is just the basic Stack Monitoring UI that ships with Kibana. I would eventually like to create my own custom dashboards once I get all the data over and indexed appropriately.

[shaunak (Shaunak Kashyap)]

No worries, got it now, thanks.

So you're monitoring an Elasticsearch cluster, say cluster A. From your OP it sounds like you do not have a separate Elasticsearch cluster just to hold your monitoring data. This setup is totally fine. You should be able to monitor cluster A and index its monitoring data back into itself. The Stack Monitoring UI in Kibana should still work with such a setup.

I have been unable to succeed at this as I have not seen data and have it sent over to Kibana.

What sort of failure are you seeing, and where are you seeing these failures?

I also was curious about the output.elasticsearch setting for each node, is this where I put each ES node or can I set a single node for all nodes to connect to?

Either one works. I'd suggest putting a few ES nodes in the list, in case one of them goes away for some reason. Metricbeat will round-robin through the list in case it needs to failover. See Configure the Elasticsearch output | Metricbeat Reference [8.11] | Elastic.

[bkasrai]

It was a combination of SSL related errors and module start up failures (assuming it had to be related to being unable to communicate with each other) I just went ahead and started a fresh configuration and now knowing what needs to be done based off of what you provided, I can attempt this and see if that rectifies the error. I will post back here shortly with an update. Thanks again @shaunak for your prompt response and attention to this matter.