Kibana stack monitoring all nodes reported to be master

Thuunder7 · July 25, 2022, 2:47pm

Hello,

I recently upgraded my beats, from 7.16.3 to 8.2.3 and after upgrading, for some reason metricbeat is reporting that all elasticsearch nodes are master.

Additionally metricbeat is also reporting a wrong elasticsearch version.

Is this a bug?
Regards

cheiligers · July 25, 2022, 8:31pm

@Thuunder7 did you upgrade your whole stack to 8.2.3? what version is your Elasticsearch and Kibana running on? The metricbeat UI in Kibana shows the data stored in your metrics-* elasticsearch indices and if the documents report the same data as what you're seeing in the UI, then there might be something else going on.

Thuunder7 · July 27, 2022, 6:31pm

Yes i did upgrade whole stack to 8.2.3. First i upgraded elasticsearch nodes and kibana but i gave two weeks until i decided to upgrade metricbeat and logstash. During those 2 weeks with metricbeat being on version 7.16.3 everything looked fine on Kibana stack monitoring, but when i upgraded metricbeat this "issue" appeared.

I looked into metrics-* indices but i only have these two:

And as you can see they both have no documents.

cheiligers · July 27, 2022, 7:02pm

@Thuunder7 I've asked the monitoring team to try and help here.

miltonhultgren · July 28, 2022, 6:24am

@Thuunder7 We've fixed these two bugs in Kibana 8.3!

You can find the related PRs below:

github.com/elastic/kibana

[Stack Monitoring] Fix displaying ES version for external collection

elastic:main ← miltonhultgren:126741-sm-wrong-version

opened 07:43PM - 28 Apr 22 UTC

miltonhultgren

+10 -1

fixes #126741 When using external collection, we were trying to read the Elas…ticsearch version from `elasticsearch.version`. However, according to the [module docs](https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-elasticsearch.html) the right field is `elasticsearch.cluster.stats.state.version` (which is aliased for `cluster_state.version`). But, it seems the Metricbeat module does not actually populate this field, and it might be because the ES APIs aren't so clear about it. `elasticsearch.version` is populated from the results of `/_cluster/state` which reports `version` as a simple integer. The response from `/_cluster/stats` reports the version under both `indices` and `nodes`, both as lists. The response from `/` reports it under `version.number` but with the build type included (8.3.0-SNAPSHOT). The version shows up in a few places but not as clearly as the old internal collection documents. As a side question, what is the version of a distributed system? A lot of the version fields in the new API response are arrays since it might be the state during a rollover to contain multiple versions. In this PR, I'm resorting to using the first item in `elasticsearch.cluster.stats.nodes.versions` as the version to display in the overview.

system · August 25, 2022, 6:24am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.