Metricbeat data not fully visible in elasticsearch

[Atul_Chadha]

I am trying to use metricbeat v 7.10.2 ( ELS on same version ) and not able to see all the data for Elasticsearch module. I understand the data is dependent on master node for few metrics so i have configured it individually on all nodes.

- module: elasticsearch
  metricsets:
  - ccr
  - node_stats
  - enrich
  - index
  - index_recovery
  - index_summary
  - ml_job
  - node
  - node_stats
  - pending_tasks
  - shard
  hosts: ["http://xxx:9200"]
  username: xxx
  password: xxx
  enabled: true
  period: 120s
  scope: node
output.elasticsearch:
  hosts: [xxx]
  username: xxx
  password: xxx
metricbeat.config.modules.path: /etc/metricbeat/modules.d/*.yml
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0644

Kibana UI

Log entry from metricbeat

Both xpack and Elasticsearch modules are enabled however i am not seeing data related to unassigned shards. This is getting sent from metricbeat ( confirmed by metricbeat test modules and /var/log/metricbeat/metricbeat logs ) However its not showing up on Kibana UI. Any idea what i am missing ?

[stephenb (Stephen Brown)]

Did You try and go and refresh the index pattern and then go back to Discover and force a reload of that page and then see if the fields show up.

[Atul_Chadha]

Thanks for the response @stephenb . The refresh did increase the count of fields however i am still seeing same number of fields for shard data. Any other suggestions ?

[stephenb (Stephen Brown)]

Ahhh IC I think you enabled the wrong module in metricbeat AND then you should be looking at the Stack Monitoring App AND it creates and this create a special index that looks something like

.monitoring-es-7-mb-2021.11.29

Follow these instructions.

Collecting Elasticsearch monitoring data with Metricbeat | Elasticsearch...

note it is elasticsearch-xpack.yml module

metricbeat modules enable elasticsearch-xpack

NOT

metricbeat modules enable elasticsearch

Also BTW I did just the plain Elasticsearch module and the shard state is there as well I am not sure where you are getting that list of fields but the the shard state is in a field named

elasticsearch.shard.state

[Atul_Chadha]

Awesome! I was unaware there is a hidden index too Appreciate your quick response.

[Atul_Chadha]

@stephenb I just realized that the field is not getting mapped properly. I see it in the details for the specific doc however its not reflected in the index pattern ( tried with * and specific pattern for the index )

Missing in index pattern

Per my understanding the field should have been part of the index pattern once i refreshed the field list, didnt happen as such though.

Due to this i am not able to aggregate and report. Any idea if i am doing something wrong ?

[stephenb (Stephen Brown)]

Hi @Atul_Chadha

First you should name your index pattern

.monitoring-es-7-mb-* not the full name probably what happened is it rolled over to the new data and the index pattern is not looking at the correct indices.

2nd Did you go check in the actual monitoring App if you could see the index / shards?

3rd In 7.10 You definitely need to refresh the index pattern and then force a reload in the Discover.

That process is more realtime in newer releases... much better.