# Metricbeat data not fully visible in elasticsearch **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [November 27, 2021, 5:19pm UTC](https://discuss.elastic.co/t/metricbeat-data-not-fully-visible-in-elasticsearch/290340 "2021-11-27T17:19:51Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Atul\_Chadha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atul_chadha/32/75469_2.png) [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)\ **Post date:** [November 27, 2021, 5:19pm UTC](https://discuss.elastic.co/t/metricbeat-data-not-fully-visible-in-elasticsearch/290340/1 "2021-11-27T17:19:51Z")
I am trying to use metricbeat v 7.10.2 ( ELS on same version ) and not able to see all the data for Elasticsearch module. I understand the data is dependent on master node for few metrics so i have configured it individually on all nodes. ```auto - module: elasticsearch metricsets: - ccr - node_stats - enrich - index - index_recovery - index_summary - ml_job - node - node_stats - pending_tasks - shard hosts: ["http://xxx:9200"] username: xxx password: xxx enabled: true period: 120s scope: node output.elasticsearch: hosts: [xxx] username: xxx password: xxx metricbeat.config.modules.path: /etc/metricbeat/modules.d/*.yml logging.level: debug logging.to_files: true logging.files: path: /var/log/metricbeat name: metricbeat keepfiles: 7 permissions: 0644 ``` Kibana UI ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f6c0bef46c0f561bac3b444abe2fb6a24c4089dc.png) Log entry from metricbeat ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/5/15b0fdc5f730666b7020257debb0446db7540450.png) Both xpack and Elasticsearch modules are enabled however i am not seeing data related to unassigned shards. This is getting sent from metricbeat ( confirmed by metricbeat test modules and /var/log/metricbeat/metricbeat logs ) However its not showing up on Kibana UI. Any idea what i am missing ? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 28, 2021, 5:38pm UTC](https://discuss.elastic.co/t/metricbeat-data-not-fully-visible-in-elasticsearch/290340/2 "2021-11-28T17:38:44Z")
Did You try and go and refresh the index pattern and then go back to Discover and force a reload of that page and then see if the fields show up. ---
**Author:** ![Atul\_Chadha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atul_chadha/32/75469_2.png) [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)\ **Post date:** [November 29, 2021, 5:34am UTC](https://discuss.elastic.co/t/metricbeat-data-not-fully-visible-in-elasticsearch/290340/3 "2021-11-29T05:34:13Z")
Thanks for the response @stephenb . The refresh did increase the count of fields however i am still seeing same number of fields for shard data. Any other suggestions ? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 29, 2021, 6:00am UTC](https://discuss.elastic.co/t/metricbeat-data-not-fully-visible-in-elasticsearch/290340/4 "2021-11-29T06:00:55Z")
Ahhh IC I think you enabled the wrong module in metricbeat AND then you should be looking at the Stack Monitoring App AND it creates and this create a special index that looks something like `.monitoring-es-7-mb-2021.11.29 ` Follow these instructions. > **[Collecting Elasticsearch monitoring data with Metricbeat | Elasticsearch...](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/configuring-metricbeat.html)** note it is `elasticsearch-xpack.yml ` module `metricbeat modules enable elasticsearch-xpack` NOT `metricbeat modules enable elasticsearch` ![Screen Shot 2021-11-28 at 9.56.03 PM](https://us1.discourse-cdn.com/elastic/original/3X/0/6/064060312ee4d6b3bbcc05c2e97d79f77112004b.png) Also BTW I did just the plain Elasticsearch module and the shard state is there as well I am not sure where you are getting that list of fields but the the shard state is in a field named `elasticsearch.shard.state` ![Screen Shot 2021-11-28 at 9.59.16 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8c3c79d4343cb0335e76c92fc9afeee5cf7cbb73.jpeg) ---
**Author:** ![Atul\_Chadha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atul_chadha/32/75469_2.png) [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)\ **Post date:** [November 29, 2021, 7:19am UTC](https://discuss.elastic.co/t/metricbeat-data-not-fully-visible-in-elasticsearch/290340/5 "2021-11-29T07:19:17Z")
Awesome! I was unaware there is a hidden index too 😀 Appreciate your quick response. ---
**Author:** ![Atul\_Chadha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atul_chadha/32/75469_2.png) [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)\ **Post date:** [December 1, 2021, 7:59am UTC](https://discuss.elastic.co/t/metricbeat-data-not-fully-visible-in-elasticsearch/290340/6 "2021-12-01T07:59:31Z")
@stephenb I just realized that the field is not getting mapped properly. I see it in the details for the specific doc however its not reflected in the index pattern ( tried with \* and specific pattern for the index ) ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/8/8894ce151830baa0f03c34b357efe1075c73d5cb.png) Missing in index pattern ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/a/bafc56c851bb2c0a628a49bda512e8ec48d0e5e2.png) Per my understanding the field should have been part of the index pattern once i refreshed the field list, didnt happen as such though. Due to this i am not able to aggregate and report. Any idea if i am doing something wrong ? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 1, 2021, 5:55pm UTC](https://discuss.elastic.co/t/metricbeat-data-not-fully-visible-in-elasticsearch/290340/7 "2021-12-01T17:55:43Z")
Hi @Atul_Chadha First you should name your index pattern `.monitoring-es-7-mb-*` not the full name probably what happened is it rolled over to the new data and the index pattern is not looking at the correct indices. 2nd Did you go check in the actual monitoring App if you could see the index / shards? 3rd In 7.10 You definitely need to refresh the index pattern and then force a reload in the Discover. That process is more realtime in newer releases... much better. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 29, 2021, 7:55pm UTC](https://discuss.elastic.co/t/metricbeat-data-not-fully-visible-in-elasticsearch/290340/8 "2021-12-29T19:55:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.