# Elasticsearch 5.5 - Deleted document count above 50%, and low disk watermark consistently hit

**URL:** <https://discuss.elastic.co/t/elasticsearch-5-5-deleted-document-count-above-50-and-low-disk-watermark-consistently-hit/138596>\
**Category:** Elasticsearch\
**Created:** [July 4, 2018, 4:28pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-deleted-document-count-above-50-and-low-disk-watermark-consistently-hit/138596 "2018-07-04T16:28:28Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![dendle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dendle/32/8629_2.png) [@dendle](https://discuss.elastic.co/u/dendle)\
**Post date:** [July 4, 2018, 4:28pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-deleted-document-count-above-50-and-low-disk-watermark-consistently-hit/138596/1 "2018-07-04T16:28:28Z")

</div>

Hello All,

I believe I have a storage issue with regards to deleted documents.

I was hoping to leave elasticsearch/lucene do its thing and merge out deletes, but i'm not sure why this isnt happening. Its only a problem as I am now consistently hitting my low disk watermark, meaning replicas will not be assigned.

I've done a force merge expunge deleted, but this is painfully slow (I have a 325GB index, its taken 13 hours and still going (in tasks API).

As to our use case, We have about 15 types, some we update often (Like diary events, think calendar), others are much more static.

Any recommendations where I can look to help me out?

Thanks for your help in advance,  
Kind Regards,  
Matt

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 4, 2018, 11:50pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-deleted-document-count-above-50-and-low-disk-watermark-consistently-hit/138596/2 "2018-07-04T23:50:31Z")

</div>

What version are you on?

---

<div class="post-metadata">

**Author:** ![dendle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dendle/32/8629_2.png) [@dendle](https://discuss.elastic.co/u/dendle)\
**Post date:** [July 5, 2018, 10:53am UTC](https://discuss.elastic.co/t/elasticsearch-5-5-deleted-document-count-above-50-and-low-disk-watermark-consistently-hit/138596/3 "2018-07-05T10:53:10Z")

</div>

Sorry, I put the version in the title but didnt mention it in the body.

I am on version 5.5 - (the incarnation before they back-ported the fix for the amazon S3 errors).

Thanks for any help

Matt

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 5, 2018, 10:58am UTC](https://discuss.elastic.co/t/elasticsearch-5-5-deleted-document-count-above-50-and-low-disk-watermark-consistently-hit/138596/4 "2018-07-05T10:58:36Z")

</div>

If possible it might be beneficial to break out types that are updated frequently into separate indices. That should mean that less data is being merged frequently, which I would expect to speed the process up.

---

<div class="post-metadata">

**Author:** ![dendle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dendle/32/8629_2.png) [@dendle](https://discuss.elastic.co/u/dendle)\
**Post date:** [July 5, 2018, 1:32pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-deleted-document-count-above-50-and-low-disk-watermark-consistently-hit/138596/5 "2018-07-05T13:32:54Z")

</div>

Thank you for your reply - whilst it is certainly not easy for our application to change in that way, I will see if it is something we can entertain. Thank you for this.

Meanwhile, are there any particular cases where segment merging would _not_ take place? The rate of growth I am seeing means that low watermark is always hit. I have read that if deleted doc count is less than 10%, it wont bother (perfectly understandable and fine). Are there any other heuristics that I need to account for? Or any features that I could have enabled/disabled that mean merging takes place less often?

Thanks again,  
Matt

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 2, 2018, 1:32pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-deleted-document-count-above-50-and-low-disk-watermark-consistently-hit/138596/6 "2018-08-02T13:32:54Z")

</div>

Closed on August 2, 2018, 1:32pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
