# Problems monitoring cluster with metricbeat on 8.15

**URL:** <https://discuss.elastic.co/t/problems-monitoring-cluster-with-metricbeat-on-8-15/365827>\
**Category:** Beats\
**Tags:** elastic-stack-monitoring, metricbeat\
**Created:** [August 30, 2024, 1:50pm UTC](https://discuss.elastic.co/t/problems-monitoring-cluster-with-metricbeat-on-8-15/365827 "2024-08-30T13:50:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![numpty-boy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/numpty-boy/32/137176_2.png) [@numpty-boy](https://discuss.elastic.co/u/numpty-boy)\
**Post date:** [August 30, 2024, 1:50pm UTC](https://discuss.elastic.co/t/problems-monitoring-cluster-with-metricbeat-on-8-15/365827/1 "2024-08-30T13:50:15Z")

</div>

Hi Team,

I've just migrated a 7.17.23 _TEST_ cluster on which stack monitoring via metricbeat was working fine, to 8.15.

Now stack monitoring has stopped working and I get taken to the 'no monitoring data found'.

If I click on metricbeat setup button, I get here:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/f/df16611771d1c2e174d93b7da4e5495a1b762918.png)

I tried running through the usual setup process [as described via page above], but no luck. Out of desperation, I removed and re-installed 8.15 versions on Kibana and Metricbeat, but again, no dice.

I'm fairly sure that we have a good connection from Metricbeat to Elasticsearch:

```auto
root@es8-3:/var/log/metricbeat# grep established 
{"log.level":"info","@timestamp":"2024-08-30T14:27:15.113+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://ES8-3:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-30T14:27:15.216+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://ES8-1:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-30T14:27:15.257+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://ES8-2:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
root@es8-3:/var/log/metricbeat#

```

... and it looks like data is being generated:

```auto
{"log.level":"info","@timestamp":"2024-08-30T14:40:32.118+0100","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":64946176}}}},"cpu":{"system":{"ticks":760,"time":{"ms":30}},"total":{"ticks":2620,"time":{"ms":100},"value":2620},"user":{"ticks":1860,"time":{"ms":70}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":18},"info":{"ephemeral_id":"bb881fc2-023a-4573-b53c-59b2068ecedb","uptime":{"ms":810098},"version":"8.15.0"},"memstats":{"gc_next":58396640,"memory_alloc":36099480,"memory_total":160672760,"rss":144990208},"runtime":{"goroutines":70}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":4,"active":0,"batches":2,"total":4},"read":{"bytes":404,"errors":2},"write":{"bytes":5113,"latency":{"histogram":{"count":49,"max":93,"mean":33.775510204081634,"median":30,"min":19,"p75":32,"p95":86.5,"p99":93,"p999":93,"stddev":15.919780157084325}}}},"pipeline":{"clients":9,"events":{"active":0,"published":3,"total":3},"queue":{"max_events":0}}},"metricbeat":{"elasticsearch":{"node_stats":{"events":3,"success":3}}},"system":{"load":{"1":0.22,"15":0.1,"5":0.13,"norm":{"1":0.0367,"15":0.0167,"5":0.0217}}}},"ecs.version":"1.6.0"}}

```

but the monitoring screen remains as in the snapshot above.

I know I must be doing something really dumb here, because I can't find posting where anyone else is having this problem.

Any suggestions most gratefully received.

Many thanks

ChIP

---

<div class="post-metadata">

**Author:** ![numpty-boy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/numpty-boy/32/137176_2.png) [@numpty-boy](https://discuss.elastic.co/u/numpty-boy)\
**Post date:** [September 3, 2024, 2:39pm UTC](https://discuss.elastic.co/t/problems-monitoring-cluster-with-metricbeat-on-8-15/365827/2 "2024-09-03T14:39:36Z")

</div>

Hi again,

I think I've definitely tied this down to the 8.15 upgrade.

I've reinstalled from scratch to 7.17.23 and set up metricbeat to monitor the cluster. Everything is fine.

I've then upgraded Elasticsearch and my standalone kibana machine to 8.15 but left metricbeat on 7.17.23. Also fine.

I then update one of the machines to metricbeat 8.15 and get this:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8f705a5b646413ea3c9b5cefbbefed682e7dc701.png)

No changes to the metricbeat config, other than to add:

```auto
output.elasticsearch:
    preset: balanced

```

as discussed in the release notes.

Here's metricbeat.yml:

```auto
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana:
  host: "Kibana.999.co.uk:5601"
  username: "elastic"
  password: "999"
  protocol: "https"
  ssl.verification_mode: none

output.elasticsearch:
  hosts: ["https://ES8-1.999.co.uk:9200","https://ES8-2.999.co.uk:9200","https://ES8-3.999.co.uk:9200"]
  username: "elastic"
  password: "999"
  protocol: "https"
  ssl.verification_mode: none
  preset: balanced

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

```

Here's modules.d/elasticsearch-xpack.yml:

```auto

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://ES8-1.999.co.uk:9200","https://ES8-2.999.co.uk:9200","https://ES8-3.999.co.uk:9200"]
  username: "elastic"
  password: "999"
  protocol: "https"
  ssl.verification_mode: none

```

Thanks again for a clue.

ChIP

---

<div class="post-metadata">

**Author:** ![numpty-boy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/numpty-boy/32/137176_2.png) [@numpty-boy](https://discuss.elastic.co/u/numpty-boy)\
**Post date:** [September 5, 2024, 4:11pm UTC](https://discuss.elastic.co/t/problems-monitoring-cluster-with-metricbeat-on-8-15/365827/3 "2024-09-05T16:11:30Z")

</div>

Hi again,

Just in case anyone else ever gets in this mess ...

I eventually got this working by

- removing metricbeat from all nodes

- removing all pre-existing metricbeat related indices from the 8.15 cluster, except for .ds-metricbeat-8.15.0-\* and .ds-.monitoring-es-8-mb-\*, including templates, etc.

- Reinstalling metric beat using the same config as before

- restarting metricbeat for all nodes

That got the stats up and working:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/1/718a739da9166c9a6563f5c49eedd144cc672d00.png)

The only problem was that the 'Ingest Pipeline' dashboard was missing, along with all other Elasticsearch related dashboards. I got around this by exporting them from an working 8.15 setup, and re-importing them.

Interestingly, when I did this, the imported dashboards included tag info, but the pre-existing ones [including those I just installed via Metricbeat] did not. Another little mystery to dig into when I get a minute.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/4/c4c2707e82b6b15e29a15105c253780ff04feba8.png)

Hope this helps.

ChIP

---

<div class="post-metadata">

**Author:** ![amorrow](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@amorrow](https://discuss.elastic.co/u/amorrow)\
**Post date:** [November 26, 2024, 5:42pm UTC](https://discuss.elastic.co/t/problems-monitoring-cluster-with-metricbeat-on-8-15/365827/4 "2024-11-26T17:42:35Z")

</div>

I know this is an older thread, but I wanted to reach out and say thank you for figuring this problem out. I ran into the exact same issue with elasticsearch metric data not appearing after upgrading from 7.17.18 to 8.16.1.

After finishing the upgrades elasticsearch and kibana, I deleted the `.monitoring-es-8-mb` datastream and allowed it to be recreated. I still have my 7.X stats and can see the data from before and after the upgrade.

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [November 27, 2024, 10:27pm UTC](https://discuss.elastic.co/t/problems-monitoring-cluster-with-metricbeat-on-8-15/365827/5 "2024-11-27T22:27:46Z")

</div>

Some additional info --  
There is an older issue here [[Stack Monitoring] Metricbeat writes to .monitoring-\*-8-mb indice instead of datastream after 8.0 upgrade · Issue #30769 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/30769) which describes a race condition between metricbeat and the Elasticsearch master node which results in the newly created 8.x monitoring index being created without the appropriate templates and as an index instead of a datastream.

Deleting the index appears to be the current workaround as described in this thread and in the issue