Monitor elasticsearch with metricbeat, cannot get "index" metricset

ebuildy · August 5, 2020, 7:28am

I am running on kubernetes, with elastic Helm, 3 master nodes and 3 data nodes. Metricbeat elastic Helm to run metricbeat.

I configure metricbeat to monitor my 6 nodes ES cluster, and give it as elasticsearch host the master k8s service.

But metricbeat complains "cannot fetch from no-master node", and I dont get the "index" metricset.

What the correct way to get all metricsets?

ebuildy · August 5, 2020, 8:16am

Okay ! the documentation is lacking this: " Install Metricbeat on each Elasticsearch node in the production cluster".

Why metricbeat is not fetching all nodes automatically?

ebuildy · August 5, 2020, 8:21am

Finaly we move to https://github.com/justwatchcom/elasticsearch_exporter , with prom/grafana.

I really try to push elastic products in my company, but it's so hard to install compare to prometheus universe

weltenwort · August 5, 2020, 3:07pm

Hi @ebuildy,

I'm sorry to hear it's not working well for you. The architecture of requiring a beat on each node puts an emphasis on reliability. It enables monitoring of nodes even if they're not part of a cluster. This might be of particular importance to diagnose the reason why it's not able to join.

I am seeing "Install Metricbeat on each Elasticsearch node in the production cluster" in the docs. Which version of the docs are you viewing?

ebuildy · August 6, 2020, 12:14pm

I have followed the doc at https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-elasticsearch.html.

Well, we use now elasticsearch-exporter for prometheus, this requires a single instance that connect to a single node and work straightaway. And there is a grafana dashboard already (whereas metricbeat is missing a dashboard for elasticsearch metrics .....).

Also, the non-master node error is logged as "debug" level:

github.com

elastic/beats/blob/44fc8e3302138dd7bd0be8d6ac2ff6754cc63c20/metricbeat/module/elasticsearch/metricset.go#L132


	// If we're talking to a set of ES nodes directly, only collect stats from the master node so
	// we don't collect the same stats from every node and end up duplicating them.
	if m.Scope == ScopeNode {
		isMaster, err := IsMaster(m.HTTP, m.GetServiceURI())
		if err != nil {
			return false, errors.Wrap(err, "error determining if connected Elasticsearch node is master")
		}

		// Not master, no event sent
		if !isMaster {
			m.Logger().Debugf("trying to fetch %v stats from a non-master node", m.Name())
			return true, nil
		}
	}

	return false, nil
}

So it took me a lot of time to find it

weltenwort · August 7, 2020, 8:13am

True, this modules' docs are a bit terse. The reason for there not being any dashboards is that the built-in stack monitoring app provides an even more curated UI for monitoring. Have tried using the stack monitoring feature?

ebuildy · August 7, 2020, 8:23am

Yes, I tried it, really nice.

In my case, I want to monitor multiple ES clusters, with a basic licence

About installing metricbeat on every node, it's really hard, maybe, you should consider to add metricbeat as a sidecar (or at least document it) when using the official elastic Helm chart for kubernetes.

We have finally moved back to prom/grafana with elasticsearch-exporter, because it's more simple and work really fast.

weltenwort · August 11, 2020, 9:11am

Yes, that could indeed be convenient for that deployment scenario. I saw you opened an issue about it - thank you!

It's good to see you found a solution that works for you in the meantime. Thanks for sharing with the community.

Have you considered deploying Metricbeat as a daemonset on each node using its helm chart and using the "autodiscover" feature to automatically enable the Elasticsearch module where needed?

system · September 8, 2020, 9:11am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.