Monitor elasticsearch with metricbeat, cannot get "index" metricset

Elastic Stack Beats
1

I am running on kubernetes, with elastic Helm, 3 master nodes and 3 data nodes. Metricbeat elastic Helm to run metricbeat.

I configure metricbeat to monitor my 6 nodes ES cluster, and give it as elasticsearch host the master k8s service.

But metricbeat complains "cannot fetch from no-master node", and I dont get the "index" metricset.

What the correct way to get all metricsets?

2

Okay ! the documentation is lacking this: " Install Metricbeat on each Elasticsearch node in the production cluster".

Why metricbeat is not fetching all nodes automatically?

3

Finaly we move to https://github.com/justwatchcom/elasticsearch_exporter , with prom/grafana.

I really try to push elastic products in my company, but it's so hard to install compare to prometheus universe :confused:

4

Hi @ebuildy,

I'm sorry to hear it's not working well for you. The architecture of requiring a beat on each node puts an emphasis on reliability. It enables monitoring of nodes even if they're not part of a cluster. This might be of particular importance to diagnose the reason why it's not able to join.

I am seeing "Install Metricbeat on each Elasticsearch node in the production cluster" in the docs. Which version of the docs are you viewing?

5

I have followed the doc at https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-elasticsearch.html.

Well, we use now elasticsearch-exporter for prometheus, this requires a single instance that connect to a single node and work straightaway. And there is a grafana dashboard already (whereas metricbeat is missing a dashboard for elasticsearch metrics .....).

Also, the non-master node error is logged as "debug" level:

So it took me a lot of time to find it :confused:

6

True, this modules' docs are a bit terse. The reason for there not being any dashboards is that the built-in stack monitoring app provides an even more curated UI for monitoring. Have tried using the stack monitoring feature?

7

Yes, I tried it, really nice.

In my case, I want to monitor multiple ES clusters, with a basic licence :confused:

About installing metricbeat on every node, it's really hard, maybe, you should consider to add metricbeat as a sidecar (or at least document it) when using the official elastic Helm chart for kubernetes.

We have finally moved back to prom/grafana with elasticsearch-exporter, because it's more simple and work really fast.

8

Yes, that could indeed be convenient for that deployment scenario. I saw you opened an issue about it - thank you!

It's good to see you found a solution that works for you in the meantime. Thanks for sharing with the community.

Have you considered deploying Metricbeat as a daemonset on each node using its helm chart and using the "autodiscover" feature to automatically enable the Elasticsearch module where needed?