# Segment size/ merge policy for large indices **URL:** \ **Category:** Elasticsearch\ **Created:** [January 20, 2023, 7:09pm UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623 "2023-01-20T19:09:59Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![GregoryJC](https://avatars.discourse-cdn.com/v4/letter/g/c37758/32.png) [@GregoryJC](https://discuss.elastic.co/u/GregoryJC)\ **Post date:** [January 20, 2023, 7:09pm UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623/1 "2023-01-20T19:09:59Z")
I am new to Elastic, I started learning about it in depth a month ago as I moved into a position to take over the more or less unmanaged cloud install we had. We have quite a few indices with segment counts ranging from 40-220 segments per index. Per our use case these indices do not roll over and will not be setup to roll over as it is not time series data. All our indices currently use the default max segment size of 5gb. The problem with this is that deletes are never being dealt with and segment count will simply continue to increase over time. We are currently on 7.17 but will be working to upgrade to 8.x in a few months. Our max heap size is 28gb. What I am wondering, especially for our worst offenders is: How large should we/can we make segment size? Is there potentially a better way to handle the problem then doing it through increasing the segment size? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 20, 2023, 7:51pm UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623/2 "2023-01-20T19:51:33Z")
Hi @GregoryJC and welcome! > [@GregoryJC](#): > > The problem with this is that deletes are never being dealt with and segment count will simply continue to increase over time. Can you explain a bit more clearly how you've come to these conclusions? Deletes are dealt with automatically, and the segment count won't grow without bound. ---
**Author:** ![GregoryJC](https://avatars.discourse-cdn.com/v4/letter/g/c37758/32.png) [@GregoryJC](https://discuss.elastic.co/u/GregoryJC)\ **Post date:** [January 20, 2023, 8:27pm UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623/3 "2023-01-20T20:27:35Z")
From what I understand after reading about segments and merges, deletes are only taken care of when merging is done. Once segments hit the max size allowed, they can no longer be merged automatically and deletes are no longer cleaned up automatically. As the data in an index grows, more and more segments will be unable to merge, and more segments will be created. I could be wrong on this. But this is my understanding of it from everything I've read so far on how it works. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 20, 2023, 9:19pm UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623/4 "2023-01-20T21:19:24Z")
> [@GregoryJC](#): > > I could be wrong on this. But this is my understanding of it from everything I've read so far on how it works. Yes this isn't correct. Merges will clean up all sizes of segments once they accumulate enough deletes. ---
**Author:** ![GregoryJC](https://avatars.discourse-cdn.com/v4/letter/g/c37758/32.png) [@GregoryJC](https://discuss.elastic.co/u/GregoryJC)\ **Post date:** [January 20, 2023, 9:41pm UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623/5 "2023-01-20T21:41:11Z")
That behavior does not seem to match what I see. Which is segments sitting at or near the 5gb limit with deletes nearing 50% of the number of docs. ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/1/418047255f1a77b66fccf3a0a50ca15cfe6b1704.png) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 20, 2023, 9:54pm UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623/6 "2023-01-20T21:54:06Z")
That sounds about right. Once they accumulate enough deletes they will be subject to merges. These ones just don't have enough deletes for it to be worth merging them yet. ---
**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Post date:** [January 21, 2023, 12:03am UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623/7 "2023-01-21T00:03:24Z")
There is always [Force merge API | Elasticsearch Guide [8.6] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 21, 2023, 6:44am UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623/8 "2023-01-21T06:44:19Z")
> [@rugenl](#): > > There is always [Force merge API | Elasticsearch Guide [8.6] | Elastic ](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) I don't think that's a good recommendation here. From those docs: > [@](#): > > **WARNING: We recommend only force merging a read-only index (meaning the index is no longer receiving writes).** There's no need to take any action here, and doing something like a force-merge will just cause different (and harder-to-fix) problems in the long run. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 18, 2023, 6:44am UTC](https://discuss.elastic.co/t/segment-size-merge-policy-for-large-indices/323623/9 "2023-02-18T06:44:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.