# Possible bug with monitoring creating thousands of elasticsearch.index.recovery duplicate docs **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring\ **Created:** [February 3, 2023, 4:04pm UTC](https://discuss.elastic.co/t/possible-bug-with-monitoring-creating-thousands-of-elasticsearch-index-recovery-duplicate-docs/324673 "2023-02-03T16:04:07Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![sparrowt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sparrowt/32/47163_2.png) [@sparrowt](https://discuss.elastic.co/u/sparrowt)\ **Post date:** [February 3, 2023, 4:04pm UTC](https://discuss.elastic.co/t/possible-bug-with-monitoring-creating-thousands-of-elasticsearch-index-recovery-duplicate-docs/324673/1 "2023-02-03T16:04:08Z")
Since upgrading to 8.x and (via [cloud.elastic.co](http://cloud.elastic.co)) enabling Monitoring \> Logs and metrics \> Ship to a deployment, I was surprised at the volume of data arriving into the `.monitoring-es-8-mb` datastream. On investigation it seems there is a lot of duplicate data being created about index recovery which happened in the past (see below). I started by investigating the most common kinds of documents arriving in that datastream broken down by `event.dataset` and they are ~60% `elasticsearch.index.recovery`, ~35% `elasticsearch.index` and then a few little other bits. Digging into the `event.dataset`=`elasticsearch.index.recovery` ones \>98% of them have `elasticsearch.index.recovery.type`=`PEER` and basically all of them have `elasticsearch.index.recovery.stage`=`DONE` which started to seem odd - why so many docs about things which have happened but none about things in progress? Filtering for 1 specific value of `elasticsearch.index.recovery.name` (in my case for example `.ds-my.index.name-YYYY.MM.DD-00000N`) I found that there were 2 records being created **every 10 seconds** for this, with apparently no new data in them. In my case there were only 2 unique values each for `elasticsearch.index.recovery.start_time.ms` and `elasticsearch.index.recovery.stop_time.ms` so it looks like the same pair of events are being duplicated every 10 seconds. Why this seeming waste of effort space (which of course we're paying for)? Is this expected / a known issue / needs a ticket? I couldn't find anything in my search so far. Thanks in advance! P.S. looking separately at the `event.dataset`=`elasticsearch.index` (as opposed to the ...`.recovery` ones) that seems to have 1 record per index every 10 seconds - although this too seems excessive for ancient indices in the cold tier, at least I can see some use in recording potentially-changing info (e.g. `elasticsearch.index.total.search.query_time_in_millis` might change over time even for an older index) whereas the recovery ones mentioned above seem to be 100% identical records of a past event so I'm not sure those need re-recording every 10 seconds. ---
**Author:** ![sparrowt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sparrowt/32/47163_2.png) [@sparrowt](https://discuss.elastic.co/u/sparrowt)\ **Post date:** [February 6, 2023, 9:20am UTC](https://discuss.elastic.co/t/possible-bug-with-monitoring-creating-thousands-of-elasticsearch-index-recovery-duplicate-docs/324673/2 "2023-02-06T09:20:57Z")
I came across [Elasticsearch index\_recovery metricset | Metricbeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-elasticsearch-index_recovery.html) which made me wonder whether the problem is metricbeat (which is presumably the underlying thing in use here) being configured wrongly: > By default only data about indices which are under active recovery are fetched. To gather data about all indices set `index_recovery.active_only: false` so is the way [elastic.co](http://elastic.co) configures metricbeat for shipping metrics setting `index_recovery.active_only: false` perhaps unnecessarily? ---
**Author:** ![sparrowt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sparrowt/32/47163_2.png) [@sparrowt](https://discuss.elastic.co/u/sparrowt)\ **Post date:** [February 27, 2023, 10:33am UTC](https://discuss.elastic.co/t/possible-bug-with-monitoring-creating-thousands-of-elasticsearch-index-recovery-duplicate-docs/324673/3 "2023-02-27T10:33:32Z")
I'm wondering which GitHub repo to report this as an issue on, having not received a reply here - could someone point me in the right direction? I am assuming it's not a bug in metricbeat itself, just in the way that [cloud.elastic.co](http://cloud.elastic.co) hosted instances are configuring it for metric shipping. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 27, 2023, 11:21pm UTC](https://discuss.elastic.co/t/possible-bug-with-monitoring-creating-thousands-of-elasticsearch-index-recovery-duplicate-docs/324673/4 "2023-02-27T23:21:53Z")
Sorry for the delay in replying here, I am chasing this up internally. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 28, 2023, 10:12am UTC](https://discuss.elastic.co/t/possible-bug-with-monitoring-creating-thousands-of-elasticsearch-index-recovery-duplicate-docs/324673/5 "2023-02-28T10:12:54Z")
Ok this is known and there's work on it here - [Index recovery API time-based filtering · Issue #93463 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/93463) ---
**Author:** ![sparrowt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sparrowt/32/47163_2.png) [@sparrowt](https://discuss.elastic.co/u/sparrowt)\ **Post date:** [February 28, 2023, 1:41pm UTC](https://discuss.elastic.co/t/possible-bug-with-monitoring-creating-thousands-of-elasticsearch-index-recovery-duplicate-docs/324673/6 "2023-02-28T13:41:19Z")
Ah interesting, so from what's written there it sounds like the [cloud.elastic.co](http://cloud.elastic.co) config is deliberately requesting to include completed recoveries (not just active). It feels excessive to get and record this data every 10 seconds in the monitoring indices given it is not going to change. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 28, 2023, 1:41pm UTC](https://discuss.elastic.co/t/possible-bug-with-monitoring-creating-thousands-of-elasticsearch-index-recovery-duplicate-docs/324673/7 "2023-03-28T13:41:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.