# Good merge settings for interactively maintained index

**URL:** <https://discuss.elastic.co/t/good-merge-settings-for-interactively-maintained-index/19667>\
**Category:** Elasticsearch\
**Created:** [September 8, 2014, 8:54pm UTC](https://discuss.elastic.co/t/good-merge-settings-for-interactively-maintained-index/19667 "2014-09-08T20:54:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [September 8, 2014, 8:54pm UTC](https://discuss.elastic.co/t/good-merge-settings-for-interactively-maintained-index/19667/1 "2014-09-08T20:54:44Z")

</div>

My indexes change somewhat frequently. If I let leave the merge settings  
as the default I end up with 25%-40% deleted documents (some indexes  
higher, some lower). I'm looking for some generic advice on:

1. Is that 25%-40% ok?
2. What kind of settings should I set to keep that in an acceptable  
range? For some meaning of acceptable.

On (1) I'm pretty sure 25%-40% is OK for my low query traffic indexes - no  
use optimizing them anyway. But for my high search traffic indexes I  
_think_ I see a performance improvement when I have lower (\<5%) deleted  
documents and fewer segments. But computers are complicated and my  
performance tests might just have been testing cache warming.... Does this  
conclusion match other's experience?

On (2) I'm not really sure what to do. It _looks_ _like_ Lucene isn't  
picking up the bigger segments to merge the deletes out of them. I assume  
that is because they are bumping against the max allowed segment size and  
therefor it can only merge one at a time so it always has something better  
to do. I'm not sure that is healthy though. Some of those old segments  
can get really bloated - like 40%-50% deleted.

Thanks!

Nik

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3AAYSOo%3DErSEp%3Dp-DpDJw7eKZObt%2B3gEmHdFO44uwsEg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3AAYSOo%3DErSEp%3Dp-DpDJw7eKZObt%2B3gEmHdFO44uwsEg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Michal\_Taborsky](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Michal\_Taborsky](https://discuss.elastic.co/u/Michal_Taborsky)\
**Post date:** [December 4, 2014, 9:06am UTC](https://discuss.elastic.co/t/good-merge-settings-for-interactively-maintained-index/19667/2 "2014-12-04T09:06:40Z")

</div>

Hello Nikolas,

we are facing similar behavior. Did you find out anything?

Thank you,  
Michal

Dne pondělí, 8. září 2014 22:55:12 UTC+2 Nikolas Everett napsal(a):

> My indexes change somewhat frequently. If I let leave the merge settings  
> as the default I end up with 25%-40% deleted documents (some indexes  
> higher, some lower). I'm looking for some generic advice on:
> 
> 1. Is that 25%-40% ok?
> 2. What kind of settings should I set to keep that in an acceptable  
> range? For some meaning of acceptable.
> 
> On (1) I'm pretty sure 25%-40% is OK for my low query traffic indexes - no  
> use optimizing them anyway. But for my high search traffic indexes I  
> _think_ I see a performance improvement when I have lower (\<5%) deleted  
> documents and fewer segments. But computers are complicated and my  
> performance tests might just have been testing cache warming.... Does this  
> conclusion match other's experience?
> 
> On (2) I'm not really sure what to do. It _looks_ _like_ Lucene isn't  
> picking up the bigger segments to merge the deletes out of them. I assume  
> that is because they are bumping against the max allowed segment size and  
> therefor it can only merge one at a time so it always has something better  
> to do. I'm not sure that is healthy though. Some of those old segments  
> can get really bloated - like 40%-50% deleted.
> 
> Thanks!
> 
> Nik

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [December 4, 2014, 10:30am UTC](https://discuss.elastic.co/t/good-merge-settings-for-interactively-maintained-index/19667/3 "2014-12-04T10:30:38Z")

</div>

25-40% is definitely "normal" for an index where many docs are being  
replaced; I've seen this go up to ~65% before large merges bring it back  
down.

On 2) there may be some improvements we can make to Lucene default  
TieredMergePolicy here, to reclaim deletes for the "too large" segments ...  
I'll have a look.

Mike McCandless

[http://blog.mikemccandless.com](http://blog.mikemccandless.com)

On Thu, Dec 4, 2014 at 4:06 AM, Michal Taborsky [michal.taborsky@gmail.com](mailto:michal.taborsky@gmail.com)  
wrote:

> Hello Nikolas,
> 
> we are facing similar behavior. Did you find out anything?
> 
> Thank you,  
> Michal
> 
> Dne pondělí, 8. září 2014 22:55:12 UTC+2 Nikolas Everett napsal(a):
> 
> > My indexes change somewhat frequently. If I let leave the merge settings  
> > as the default I end up with 25%-40% deleted documents (some indexes  
> > higher, some lower). I'm looking for some generic advice on:
> > 
> > 1. Is that 25%-40% ok?
> > 2. What kind of settings should I set to keep that in an acceptable  
> > range? For some meaning of acceptable.
> > 
> > On (1) I'm pretty sure 25%-40% is OK for my low query traffic indexes -  
> > no use optimizing them anyway. But for my high search traffic indexes I  
> > _think_ I see a performance improvement when I have lower (\<5%) deleted  
> > documents and fewer segments. But computers are complicated and my  
> > performance tests might just have been testing cache warming.... Does this  
> > conclusion match other's experience?
> > 
> > On (2) I'm not really sure what to do. It _looks_ _like_ Lucene isn't  
> > picking up the bigger segments to merge the deletes out of them. I assume  
> > that is because they are bumping against the max allowed segment size and  
> > therefor it can only merge one at a time so it always has something better  
> > to do. I'm not sure that is healthy though. Some of those old segments  
> > can get really bloated - like 40%-50% deleted.
> > 
> > Thanks!
> > 
> > Nik
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAD7smRetuA9UDMMqPi9pZuGqUtdGGxrZM5ugP%2BVO3SVCUxTD6g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAD7smRetuA9UDMMqPi9pZuGqUtdGGxrZM5ugP%2BVO3SVCUxTD6g%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [December 4, 2014, 9:37pm UTC](https://discuss.elastic.co/t/good-merge-settings-for-interactively-maintained-index/19667/4 "2014-12-04T21:37:19Z")

</div>

OK I ran a quick test using Wikipedia docs; net/net I think  
TieredMergePolicy's (the default) behavior is fine. Once a too-large  
segment has \> 50% deletes it is eligible for merging and will be  
aggressively merged.

To visualize this, I first built a 33.3M doc Wikipedia index (append  
only), then ran forever randomly replacing each doc, which is a worst  
case test since every update also deletes a previous doc.

I set max merged segment size to 800 MB, so I had a good number (17)  
of them; otherwise I left TMP at defaults.

I refreshed every 3 seconds, and plotted the resulting graph of %tg  
deleted but not yet merge docs over time:

It quickly ramps up from 0 at the start and only falls again once  
the too-large segments start being merged and eventually stabilizes  
to a fairly narrow range of 33%-45%.

Mike McCandless

[http://blog.mikemccandless.com](http://blog.mikemccandless.com)

On Thu, Dec 4, 2014 at 5:30 AM, Michael McCandless [mike@elasticsearch.com](mailto:mike@elasticsearch.com)  
wrote:

> 25-40% is definitely "normal" for an index where many docs are being  
> replaced; I've seen this go up to ~65% before large merges bring it back  
> down.
> 
> On 2) there may be some improvements we can make to Lucene default  
> TieredMergePolicy here, to reclaim deletes for the "too large" segments ...  
> I'll have a look.
> 
> Mike McCandless
> 
> [http://blog.mikemccandless.com](http://blog.mikemccandless.com)
> 
> On Thu, Dec 4, 2014 at 4:06 AM, Michal Taborsky \<[michal.taborsky@gmail.com](mailto:michal.taborsky@gmail.com)
> 
> > wrote:
> 
> > Hello Nikolas,
> > 
> > we are facing similar behavior. Did you find out anything?
> > 
> > Thank you,  
> > Michal
> > 
> > Dne pondělí, 8. září 2014 22:55:12 UTC+2 Nikolas Everett napsal(a):
> > 
> > > My indexes change somewhat frequently. If I let leave the merge  
> > > settings as the default I end up with 25%-40% deleted documents (some  
> > > indexes higher, some lower). I'm looking for some generic advice on:
> > > 
> > > 1. Is that 25%-40% ok?
> > > 2. What kind of settings should I set to keep that in an acceptable  
> > > range? For some meaning of acceptable.
> > > 
> > > On (1) I'm pretty sure 25%-40% is OK for my low query traffic indexes -  
> > > no use optimizing them anyway. But for my high search traffic indexes I  
> > > _think_ I see a performance improvement when I have lower (\<5%) deleted  
> > > documents and fewer segments. But computers are complicated and my  
> > > performance tests might just have been testing cache warming.... Does this  
> > > conclusion match other's experience?
> > > 
> > > On (2) I'm not really sure what to do. It _looks_ _like_ Lucene isn't  
> > > picking up the bigger segments to merge the deletes out of them. I assume  
> > > that is because they are bumping against the max allowed segment size and  
> > > therefor it can only merge one at a time so it always has something better  
> > > to do. I'm not sure that is healthy though. Some of those old segments  
> > > can get really bloated - like 40%-50% deleted.
> > > 
> > > Thanks!
> > > 
> > > Nik
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/faec06a2-c352-4e3e-bea0-41ace2b35d6f%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAD7smRe\_cN%2B2PtNT68z%2B5%3DDJ4W-vaO4-pUJ3bo1o0AFe%3D-4B1Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAD7smRe_cN%2B2PtNT68z%2B5%3DDJ4W-vaO4-pUJ3bo1o0AFe%3D-4B1Q%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:45am UTC](https://discuss.elastic.co/t/good-merge-settings-for-interactively-maintained-index/19667/5 "2017-07-06T00:45:46Z")

</div>