Stack Monitoring Not Stable

Elastic Stack Elasticsearch
1

I've just upgraded our cluster from 7.6.2 to 7.7.1. I've a 10 Node cluster, self monitored with metricbeat + logstash and kibana monitoring using "Stack Monitoring" feature. It worked quite well with 7.6.2 but with 7.7.1 I can't see the logstash metrics anymore in the "Stack Monitoring" tab even though the data is shipped to our dedicated monitoring cluster.

After the upgrade I also had to reset to see the Kibana and Elasticsearch metrics again:

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

It's annoying that one has do debug such things after just a minor upgrade with almost no informations in the logfiles.

Seem like Kibana knows that there is data, but it's not showing any nodes:

2020-06-08_113550
2020-06-08_113550707×513 27.3 KB

2

still the same issue with 7.8. Any hints?

3

Hi @tomx1,

Do you see any .monitoring-logstash-* indices on the monitoring cluster with recent dates? Do you Logstash monitoring data on the cluster overview page? Are you using Metricbeat to monitor Logstash?

4

Yes i do - all indices are present and fresh at the monitoring cluster. I think i got hit by this bug: Upgrade 7.6->7.7 unable to verify the first certificate"

However, my Cluster is running in docker swarm, the suggested workaround to pipe the certificate directly into the kibana.yaml does not work with this setup. And yes, we are using metricbeat to gather metrics from kibana, elasticsearch and logstash.