Hi Team,
I've just migrated a 7.17.23 TEST cluster on which stack monitoring via metricbeat was working fine, to 8.15.
Now stack monitoring has stopped working and I get taken to the 'no monitoring data found'.
If I click on metricbeat setup button, I get here:
I tried running through the usual setup process [as described via page above], but no luck. Out of desperation, I removed and re-installed 8.15 versions on Kibana and Metricbeat, but again, no dice.
I'm fairly sure that we have a good connection from Metricbeat to Elasticsearch:
root@es8-3:/var/log/metricbeat# grep established
{"log.level":"info","@timestamp":"2024-08-30T14:27:15.113+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://ES8-3:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-30T14:27:15.216+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://ES8-1:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-30T14:27:15.257+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://ES8-2:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
root@es8-3:/var/log/metricbeat#
... and it looks like data is being generated:
{"log.level":"info","@timestamp":"2024-08-30T14:40:32.118+0100","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":64946176}}}},"cpu":{"system":{"ticks":760,"time":{"ms":30}},"total":{"ticks":2620,"time":{"ms":100},"value":2620},"user":{"ticks":1860,"time":{"ms":70}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":18},"info":{"ephemeral_id":"bb881fc2-023a-4573-b53c-59b2068ecedb","uptime":{"ms":810098},"version":"8.15.0"},"memstats":{"gc_next":58396640,"memory_alloc":36099480,"memory_total":160672760,"rss":144990208},"runtime":{"goroutines":70}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":4,"active":0,"batches":2,"total":4},"read":{"bytes":404,"errors":2},"write":{"bytes":5113,"latency":{"histogram":{"count":49,"max":93,"mean":33.775510204081634,"median":30,"min":19,"p75":32,"p95":86.5,"p99":93,"p999":93,"stddev":15.919780157084325}}}},"pipeline":{"clients":9,"events":{"active":0,"published":3,"total":3},"queue":{"max_events":0}}},"metricbeat":{"elasticsearch":{"node_stats":{"events":3,"success":3}}},"system":{"load":{"1":0.22,"15":0.1,"5":0.13,"norm":{"1":0.0367,"15":0.0167,"5":0.0217}}}},"ecs.version":"1.6.0"}}
but the monitoring screen remains as in the snapshot above.
I know I must be doing something really dumb here, because I can't find posting where anyone else is having this problem.
Any suggestions most gratefully received.
Many thanks
ChIP
