# Limiting Segment Memory Consumption **URL:** \ **Category:** Elasticsearch\ **Created:** [April 13, 2016, 10:35pm UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319 "2016-04-13T22:35:14Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![atul](https://avatars.discourse-cdn.com/v4/letter/a/f14d63/32.png) [@atul](https://discuss.elastic.co/u/atul)\ **Post date:** [April 13, 2016, 10:35pm UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/1 "2016-04-13T22:35:14Z")
We have been running under constant memory pressure on our ES nodes. Upon analyzing it appears that segment is consuming more than 50% of the available heap. Is there a way to configure the amount of memory that can be used for caching segment data? We though about reducing shards, but then that would lead to very large shards, our index is about 1TB. Is there a better practice / strategy to use in such cases? We are running 1.7. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 13, 2016, 11:08pm UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/2 "2016-04-13T23:08:49Z")
How many shards do you have? ---
**Author:** ![atul](https://avatars.discourse-cdn.com/v4/letter/a/f14d63/32.png) [@atul](https://discuss.elastic.co/u/atul)\ **Post date:** [April 14, 2016, 12:13am UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/3 "2016-04-14T00:13:38Z")
Currently - 24 - its coming down to about 40gb per shard. I understand it not ideal but data grew faster than expected. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 14, 2016, 12:52am UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/4 "2016-04-14T00:52:27Z")
40GB is fine. What sort of data are you dealing with? ---
**Author:** ![atul](https://avatars.discourse-cdn.com/v4/letter/a/f14d63/32.png) [@atul](https://discuss.elastic.co/u/atul)\ **Post date:** [April 14, 2016, 2:39am UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/5 "2016-04-14T02:39:28Z")
Data is quite heterogeneous - but largely its e-commerce / retail orders including customer requests with fields such as array of comments, phone number, email id, dates etc. We also have international customers and so some non-english characters (largely east european) in there. I checked the .tim files on the disk and they added up to 40gb. For shard size - i thought 40 gb was crossing the "reasonable" limit as suggested in a few online posts. Is there such a limit for a shard where query latency and / or shard replication / relocation is severely hampered? ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [April 14, 2016, 3:30am UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/6 "2016-04-14T03:30:46Z")
If I remember correctly, the somewhat ideal shard size is under 50GB from a presentation by @warkolm ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 14, 2016, 3:54am UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/7 "2016-04-14T03:54:08Z")
And you're putting this all into a single index? Sounds like you need to break it up a bit more, it only time based by order time. > [@atul](#): > > I checked the .tim files on the disk and they added up to 40gb That's wrong, use the APIs. ---
**Author:** ![atul](https://avatars.discourse-cdn.com/v4/letter/a/f14d63/32.png) [@atul](https://discuss.elastic.co/u/atul)\ **Post date:** [April 14, 2016, 4:11am UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/8 "2016-04-14T04:11:14Z")
I did use the api but couldn't find if the term dictionary size is exposed by the api. The memory consumed by segments was from the api. We are evaluating whether time based could work in our functional scenario. On the original question though - is there a way to limit the amount of heap used by segments? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 14, 2016, 4:20am UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/9 "2016-04-14T04:20:04Z")
Can you share where you are seeing the segments using all the memory? Ie the output of the API call? ---
**Author:** ![atul](https://avatars.discourse-cdn.com/v4/letter/a/f14d63/32.png) [@atul](https://discuss.elastic.co/u/atul)\ **Post date:** [April 14, 2016, 4:28am UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/10 "2016-04-14T04:28:21Z")
Here is the snippet - segments" : { "count" : 969, "memory\_in\_bytes" : 46613585150, "index\_writer\_memory\_in\_bytes" : 5838763, "index\_writer\_max\_memory\_in\_bytes" : 8252512238, "version\_map\_memory\_in\_bytes" : 28576, "fixed\_bit\_set\_memory\_in\_bytes" : 78476616 } Appreciate your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:59pm UTC](https://discuss.elastic.co/t/limiting-segment-memory-consumption/47319/11 "2017-07-05T22:59:24Z")