# Too many Deleted Docs **URL:** \ **Category:** Elasticsearch\ **Created:** [May 8, 2017, 5:54pm UTC](https://discuss.elastic.co/t/too-many-deleted-docs/84964 "2017-05-08T17:54:05Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ABILASH](https://avatars.discourse-cdn.com/v4/letter/a/4af34b/32.png) [@ABILASH](https://discuss.elastic.co/u/ABILASH)\ **Post date:** [May 8, 2017, 5:54pm UTC](https://discuss.elastic.co/t/too-many-deleted-docs/84964/1 "2017-05-08T17:54:05Z")
Hi, We have a update heavy index with lot of nested documents. We are using elasticsearch 2.3 and I am concerned about the amount of deleted documents in the index. We are seeing upto 50% deleted documents. I have tried multiple merge settings with the Tiered Merge Policy. Here are my index settings. "index": { "codec": "best\_compression", "refresh\_interval": "300s", "number\_of\_shards": "20", "translog": { "flush\_threshold\_size": "2048mb", "durability": "async" }, "merge": { "policy": { "max\_merge\_at\_once": "20", "max\_merged\_segment": "15GB", "expunge\_deletes\_allowed": "5", "segments\_per\_tier": "20" } }, "gc\_deletes": "1s", "max\_result\_window": "200000", "requests": { "cache": { "enable": "true" } }, "uuid": "erNEn9dVQGq6dcyavK11wA" } I see that there is some merge activity going on in the cluster in the back ground using the hot threads api. But still the deleted documents are really high percentage which is occupying disk space. Please advice on what I should do to bring this number down without using the optimize / force merge ? ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [May 9, 2017, 9:12am UTC](https://discuss.elastic.co/t/too-many-deleted-docs/84964/2 "2017-05-09T09:12:39Z")
This is to be expected for a heavy-update use-case. I wouldn't worry about the percentage of deleted docs. ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [May 9, 2017, 9:16am UTC](https://discuss.elastic.co/t/too-many-deleted-docs/84964/3 "2017-05-09T09:16:43Z")
If you really really really need to get the number of deletions lower, you could set `index.merge.policy.reclaim_deletes_weight` to something like `3` (default is `2` and `3` is very high already, make sure to not go beyond `3`). This will tell Elasticsearch to favor merges that have the higher number of deleted documents. But I would not expect it to suddenly reduce the number of deleted docs magically. If you have a heavy-update case then most likely all your segments have a significant number of deletes. ---
**Author:** ![ABILASH](https://avatars.discourse-cdn.com/v4/letter/a/4af34b/32.png) [@ABILASH](https://discuss.elastic.co/u/ABILASH)\ **Post date:** [May 10, 2017, 4:28am UTC](https://discuss.elastic.co/t/too-many-deleted-docs/84964/4 "2017-05-10T04:28:11Z")
Hi @jpountz Thanks a lot for your response. Please let me know if my understanding is correct, increasing reclaim\_deletes\_weight will favor segments with more deleted docs from the documentation. For the existing segments that have already reached max\_merged\_segment , will they be affected by this setting also? The reason why I am worried about the deleted documents , is that they are taking up lot of space and I have read these awesome blogs [http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html) and [https://www.elastic.co/blog/lucenes-handling-of-deleted-documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents) by Micheal. Here are some of the reasons why I am worried about the deleted docs. We have lot of nested type fields in the elasticsearch mapping which is creating lot of documents for each primary doc we have . We currently have about 20 billion docs(about 3.5 TB with deleted docs, about 2.2 TB without deleted docs ) in the index with 20 primary shards and the data is growing at a rapid pace. I am worried about reaching the 2 billion docs per shard with lot of deleted docs. In terms of capacity planning , from @mikemccand blogs , it feels like we have to account for 1.5X disk space . One thing that I have read from the blog is that Tiered Merge Policy does not reclaim deleted documents from segments that have already reached max\_merged\_segment size. So I was just wondering for my use case, if it is better to use other merge policies like LogByteSizeMergePolicy to reduce the overall deleted documents ? But if TieredMergePolicy is the way to go, then we will account for the 1.5X disk space in terms of infrastructure. Please advice. ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [May 10, 2017, 5:02pm UTC](https://discuss.elastic.co/t/too-many-deleted-docs/84964/5 "2017-05-10T17:02:20Z")
> [@ABILASH](#): > > For the existing segments that have already reached max\_merged\_segment , will they be affected by this setting also? Only segments whose size multiplied by the ratio of live documents is less than `max_merged_segment/2` are eligible to be merged. So for instance if you have a segment of size `max_merged_segment`, it will only become eligible for merging once it reaches 50% deleted documents. > [@ABILASH](#): > > So I was just wondering for my use case, if it is better to use other merge policies like LogByteSizeMergePolicy to reduce the overall deleted documents ? No, switching to a different merge policy is unlikely to help, quite the contrary since eg. LogByteMergePolicy does not try to select merges that have more deletions. ---
**Author:** ![ABILASH](https://avatars.discourse-cdn.com/v4/letter/a/4af34b/32.png) [@ABILASH](https://discuss.elastic.co/u/ABILASH)\ **Post date:** [May 10, 2017, 5:20pm UTC](https://discuss.elastic.co/t/too-many-deleted-docs/84964/6 "2017-05-10T17:20:42Z")
Thanks @jpountz for all your inputs. Will account for the 1.5X disk space / documents since this use case is a update heavy one. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 7, 2017, 5:21pm UTC](https://discuss.elastic.co/t/too-many-deleted-docs/84964/7 "2017-06-07T17:21:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.