Monitoring an Elasticsearch Cluster with a Single Metricbeat Instance

[bunste (Stefan Bungter)]

We are currently on version 7.17.5 with the full ELK stack and are working on getting ready to jump to 8.x with the help of Upgrade Assistant.

One of the cluster issues is:

Setting [xpack.monitoring.collection.enabled] is deprecated

The Documentation link for that issue says

Using the Monitoring plugin to collect and ship monitoring data is deprecated. Metricbeat is the recommended method for collecting and shipping monitoring data to a monitoring cluster.

Ok so I took a look at Metricbeat:

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/configuring-metricbeat.html

On this page it says under point 1 that you should set xpack.monitoring.collection.enabled to true. But the issue we are coming from clearly says that this setting is deprecated! I'm slightly confused but keep reading for now....

On the same page under point 2 it says:

Ideally install a single Metricbeat instance configured with scope: cluster and configure hosts to point to an endpoint (e.g. a load-balancing proxy) which directs requests to the master-ineligible nodes in the cluster.

I would like to focus here on what the proxy should point to: to the master-ineligible nodes in the cluster.

Now let's compare this information with the section about the elasticsearch module from the Metricbeat doc:

https://www.elastic.co/guide/en/beats/metricbeat/7.17/metricbeat-module-elasticsearch.html

It says:

If scope is set to cluster, each entry in the hosts list indicates a single endpoint for a distinct Elasticsearch cluster (for example, a load-balancing proxy fronting the cluster).

There is no mention of the fact that only certain nodes may be reached via this endpoint. Perhaps this is common knowledge but I am a bit unsure and would like some clarification on the subject.

Our situation is as follows:

We have an Elasticsearch cluster with dedicated master, ingest and data nodes. To be clear, these nodes each have exactly one role: either master or ingest or data. Furthermore, there is a load-balancing proxy that distributes all traffic to the two ingest nodes. All other applications use this proxy to communicate with the cluster in some way.

Is that correct now or what should the proxy point to?

(1) Only master-ineligible nodes
(2) All master-ineligible nodes

This does make a difference. We currently have (1) fulfilled but not (2) - for that, the data nodes would also have to be behind the proxy. Which would appear wrong to me somehow.

Is our setup correct and with this setup would a single Metricbeat instance be sufficient to monitor Elasticsearch with scope: cluster?

Many thanks in advance for your help!

[matschaffer]

Hi there @bunste and welcome back!

You should be able to use a single metricbeat in scope: cluster to retrieve monitoring data from any node in the cluster. It may be fine to use the existing proxy, though that will place it in contention with application traffic. Your load balancer throughput and bandwidth would determine if this is a problem or not.

The documentation is indeed confusing. I've opened up Revise configuring-metricbeat documentation · Issue #139338 · elastic/kibana · GitHub to address that.