# ES loses the data being collected when deleting the index **URL:** \ **Category:** Elasticsearch\ **Created:** [December 11, 2018, 12:19pm UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377 "2018-12-11T12:19:50Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![horaceli](https://avatars.discourse-cdn.com/v4/letter/h/35a633/32.png) [@horaceli](https://discuss.elastic.co/u/horaceli)\ **Post date:** [December 11, 2018, 12:19pm UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/1 "2018-12-11T12:19:50Z")
es lose data when the cluster is deleting index.As shown below ![20181211](https://us1.discourse-cdn.com/elastic/original/3X/9/3/93920dde037b26b99bac3872c41b5519eef3e7b6.png) how to fix the problem? I mean, when I delete the index, the amount of data being collected is also reduced.The picture shows the amount of data collected at each time(per minute). The cluster has four nodes, one of them setting is: node.master: false node.data: false node.ingest: false the rest of the nodes setting is default. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 11, 2018, 4:09pm UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/2 "2018-12-11T16:09:52Z")
Isn't it expected that the amount of data in the cluster goes down if you delete an index? ---
**Author:** ![horaceli](https://avatars.discourse-cdn.com/v4/letter/h/35a633/32.png) [@horaceli](https://discuss.elastic.co/u/horaceli)\ **Post date:** [December 12, 2018, 1:48am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/3 "2018-12-12T01:48:21Z")
Sorry, I didn’t make it clear.I mean, when I delete the index, the amount of data being collected is also reduced.The picture shows the amount of data collected at each time(per minute). ---
**Author:** ![horaceli](https://avatars.discourse-cdn.com/v4/letter/h/35a633/32.png) [@horaceli](https://discuss.elastic.co/u/horaceli)\ **Post date:** [December 16, 2018, 6:28am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/4 "2018-12-16T06:28:29Z")
anyone can help me? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 16, 2018, 8:37am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/5 "2018-12-16T08:37:08Z")
Is there anything in the logs around that time? If indexing slows down at that time, this does generally not mean that data will be lost as clients generally will retry on failure or timeout. ---
**Author:** ![horaceli](https://avatars.discourse-cdn.com/v4/letter/h/35a633/32.png) [@horaceli](https://discuss.elastic.co/u/horaceli)\ **Post date:** [December 20, 2018, 6:55am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/6 "2018-12-20T06:55:54Z")
es logs: [2018-12-20T04:02:40,847][INFO][o.e.c.m.MetaDataDeleteIndexService] [iom] [filebeat-2018.12.12/AzYCjS0jTsy3ovBI8An13A] deleting index [2018-12-20T04:03:10,880][WARN][o.e.d.z.PublishClusterStateAction] [iom] timed out waiting for all nodes to process published state [3888] (timeout [30s], pending nodes: [{iom}{XWu3adjESaGrlDdqONE4-A}{SKwUJfhaT8GeO31NxV-Fbw}{172.31.24.86}{172.31.24.86:9300}, {iom}{ZEh8uJqGSV2EytseLQSz4w}{82xwXo0eQNmVV6aWfTXVTw}{172.31.24.88}{172.31.24.88:9300}]) [2018-12-20T04:05:10,947][DEBUG][o.e.a.a.i.d.TransportDeleteIndexAction] [iom] failed to delete indices [[[metricbeat-2018.12.12/S0asWes4TZyGJ6XCCjg76g]]] org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (delete-index [[metricbeat-2018.12.12/S0asWes4TZyGJ6XCCjg76g]]) within 2m at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$null$0(ClusterService.java:255) ~[elasticsearch-5.6.3.jar:5.6.3] at java.util.ArrayList.forEach(ArrayList.java:1255) ~[?:1.8.0\_151] at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$onTimeout$1(ClusterService.java:254) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.3.jar:5.6.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_151] at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_151] [2018-12-20T04:11:00,175][WARN][o.e.c.s.ClusterService] [iom] cluster state update task [delete-index [[filebeat-2018.12.12/AzYCjS0jTsy3ovBI8An13A]]] took [8.3m] above the warn threshold of 30s ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 20, 2018, 6:58am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/7 "2018-12-20T06:58:12Z")
What is the output off the [cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html)? ---
**Author:** ![horaceli](https://avatars.discourse-cdn.com/v4/letter/h/35a633/32.png) [@horaceli](https://discuss.elastic.co/u/horaceli)\ **Post date:** [December 20, 2018, 7:01am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/8 "2018-12-20T07:01:52Z")
Our cluster usually delete the index at 4:00 am, the above logs show that the cluster is deleting the index called filebeat-2018.12.12(because the cluster stores 7 days of the data). ---
**Author:** ![horaceli](https://avatars.discourse-cdn.com/v4/letter/h/35a633/32.png) [@horaceli](https://discuss.elastic.co/u/horaceli)\ **Post date:** [December 20, 2018, 7:02am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/9 "2018-12-20T07:02:51Z")
{ "cluster\_name": "iot-elk-cluster", "status": "green", "timed\_out": false, "number\_of\_nodes": 4, "number\_of\_data\_nodes": 3, "active\_primary\_shards": 87, "active\_shards": 175, "relocating\_shards": 0, "initializing\_shards": 0, "unassigned\_shards": 0, "delayed\_unassigned\_shards": 0, "number\_of\_pending\_tasks": 0, "number\_of\_in\_flight\_fetch": 0, "task\_max\_waiting\_in\_queue\_millis": 0, "active\_shards\_percent\_as\_number": 100 } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 20, 2018, 7:12am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/10 "2018-12-20T07:12:13Z")
It looks like it is timing out publishing the cluster state. Is the cluster under heavy load? Do you see any long or frequent GC? Do you have `minimum_master_nodes` set to `2` given that you appear to have 3 master-eligible nodes? ---
**Author:** ![horaceli](https://avatars.discourse-cdn.com/v4/letter/h/35a633/32.png) [@horaceli](https://discuss.elastic.co/u/horaceli)\ **Post date:** [December 20, 2018, 7:21am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/11 "2018-12-20T07:21:44Z")
It seems that our cluster is really heavily loaded, each ES node has 16 core CPUs, but the cpu load is above 10, and one of them is around 16. But GC is rare, the ES node is 64G memory, and the es memory configured is 31G. minimum\_master\_nodes set is 2 ---
**Author:** ![horaceli](https://avatars.discourse-cdn.com/v4/letter/h/35a633/32.png) [@horaceli](https://discuss.elastic.co/u/horaceli)\ **Post date:** [December 21, 2018, 9:01am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/12 "2018-12-21T09:01:58Z")
Does anyone has good ideas? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 21, 2018, 9:06am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/13 "2018-12-21T09:06:36Z")
It is quite possible that the cluster is overloaded and not able to process and distribute changes to the cluster state fast enough. One way to address this would be to scale out the cluster and distribute the load across more hosts. You may also benefit from introducing 3 small dedicated master nodes. These typically do not use a lot of resources, but as they do not hold data or serve traffic they do not get overloaded and can focus on managing the cluster state. ---
**Author:** ![horaceli](https://avatars.discourse-cdn.com/v4/letter/h/35a633/32.png) [@horaceli](https://discuss.elastic.co/u/horaceli)\ **Post date:** [December 21, 2018, 9:10am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/14 "2018-12-21T09:10:19Z")
thanks, i will try ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2019, 9:10am UTC](https://discuss.elastic.co/t/es-loses-the-data-being-collected-when-deleting-the-index/160377/15 "2019-01-18T09:10:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.