Problems monitoring cluster with metricbeat on 8.15

numpty-boy · August 30, 2024, 1:50pm

Hi Team,

I've just migrated a 7.17.23 TEST cluster on which stack monitoring via metricbeat was working fine, to 8.15.

Now stack monitoring has stopped working and I get taken to the 'no monitoring data found'.

If I click on metricbeat setup button, I get here:

I tried running through the usual setup process [as described via page above], but no luck. Out of desperation, I removed and re-installed 8.15 versions on Kibana and Metricbeat, but again, no dice.

I'm fairly sure that we have a good connection from Metricbeat to Elasticsearch:

root@es8-3:/var/log/metricbeat# grep established 
{"log.level":"info","@timestamp":"2024-08-30T14:27:15.113+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://ES8-3:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-30T14:27:15.216+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://ES8-1:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-30T14:27:15.257+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://ES8-2:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
root@es8-3:/var/log/metricbeat#

... and it looks like data is being generated:

{"log.level":"info","@timestamp":"2024-08-30T14:40:32.118+0100","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":64946176}}}},"cpu":{"system":{"ticks":760,"time":{"ms":30}},"total":{"ticks":2620,"time":{"ms":100},"value":2620},"user":{"ticks":1860,"time":{"ms":70}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":18},"info":{"ephemeral_id":"bb881fc2-023a-4573-b53c-59b2068ecedb","uptime":{"ms":810098},"version":"8.15.0"},"memstats":{"gc_next":58396640,"memory_alloc":36099480,"memory_total":160672760,"rss":144990208},"runtime":{"goroutines":70}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":4,"active":0,"batches":2,"total":4},"read":{"bytes":404,"errors":2},"write":{"bytes":5113,"latency":{"histogram":{"count":49,"max":93,"mean":33.775510204081634,"median":30,"min":19,"p75":32,"p95":86.5,"p99":93,"p999":93,"stddev":15.919780157084325}}}},"pipeline":{"clients":9,"events":{"active":0,"published":3,"total":3},"queue":{"max_events":0}}},"metricbeat":{"elasticsearch":{"node_stats":{"events":3,"success":3}}},"system":{"load":{"1":0.22,"15":0.1,"5":0.13,"norm":{"1":0.0367,"15":0.0167,"5":0.0217}}}},"ecs.version":"1.6.0"}}

but the monitoring screen remains as in the snapshot above.

I know I must be doing something really dumb here, because I can't find posting where anyone else is having this problem.

Any suggestions most gratefully received.

Many thanks

ChIP

numpty-boy · September 3, 2024, 2:39pm

Hi again,

I think I've definitely tied this down to the 8.15 upgrade.

I've reinstalled from scratch to 7.17.23 and set up metricbeat to monitor the cluster. Everything is fine.

I've then upgraded Elasticsearch and my standalone kibana machine to 8.15 but left metricbeat on 7.17.23. Also fine.

I then update one of the machines to metricbeat 8.15 and get this:

No changes to the metricbeat config, other than to add:

output.elasticsearch:
    preset: balanced

as discussed in the release notes.

Here's metricbeat.yml:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression


setup.kibana:
  host: "Kibana.999.co.uk:5601"
  username: "elastic"
  password: "999"
  protocol: "https"
  ssl.verification_mode: none

output.elasticsearch:
  hosts: ["https://ES8-1.999.co.uk:9200","https://ES8-2.999.co.uk:9200","https://ES8-3.999.co.uk:9200"]
  username: "elastic"
  password: "999"
  protocol: "https"
  ssl.verification_mode: none
  preset: balanced

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Here's modules.d/elasticsearch-xpack.yml:


- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://ES8-1.999.co.uk:9200","https://ES8-2.999.co.uk:9200","https://ES8-3.999.co.uk:9200"]
  username: "elastic"
  password: "999"
  protocol: "https"
  ssl.verification_mode: none

Thanks again for a clue.

ChIP

numpty-boy · September 5, 2024, 4:11pm

Hi again,

Just in case anyone else ever gets in this mess ...

I eventually got this working by

removing metricbeat from all nodes
removing all pre-existing metricbeat related indices from the 8.15 cluster, except for .ds-metricbeat-8.15.0-* and .ds-.monitoring-es-8-mb-*, including templates, etc.
Reinstalling metric beat using the same config as before
restarting metricbeat for all nodes

That got the stats up and working:

The only problem was that the 'Ingest Pipeline' dashboard was missing, along with all other Elasticsearch related dashboards. I got around this by exporting them from an working 8.15 setup, and re-importing them.

Interestingly, when I did this, the imported dashboards included tag info, but the pre-existing ones [including those I just installed via Metricbeat] did not. Another little mystery to dig into when I get a minute.

Hope this helps.

ChIP