# Search on frozen index cause huge usage of JVM Heap **URL:** \ **Category:** Elasticsearch\ **Created:** [July 31, 2019, 11:07am UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101 "2019-07-31T11:07:25Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Ali\_Reza\_Haghighat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_reza_haghighat/32/48590_2.png) [@Ali\_Reza\_Haghighat](https://discuss.elastic.co/u/Ali_Reza_Haghighat)\ **Post date:** [July 31, 2019, 11:07am UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/1 "2019-07-31T11:07:25Z")
Hi, I have run the following test to analyze the effect of using frozen indices of JVM heap usage. I have a single node cluster with the following properties: - JVM Heap: 6 GB - Number of indices: 200 - Number of shards: 200 - Number of the freezed indices: 200 (all indices are freezed!) - Size of each shard: 3 GB - Used JVM Heap: 3.1 GB I run a simple query on the Elastic: > { "query": { "match\_all": {} } } Running this query results in huge usage of JVM heap (reaching to 5.99 GB) for 3-4 minutes. Finally, Elastic was able to return the result, however, based on the Elastic documentation, I expect much less usage of JVM heap: - [Elastic Document on Frozen Indices](https://www.elastic.co/guide/en/elasticsearch/reference/master/frozen-indices.html) > Elasticsearch builds the transient data structures of each shard of a frozen index each time that shard is searched, and discards these data structures as soon as the search is complete. Because Elasticsearch does not maintain these transient data structures in memory, frozen indices consume much less heap than normal indices - [Frozen Index Issue on Github](https://github.com/elastic/elasticsearch/issues/34352): > When a search targets frozen indices, each index would be searched sequentially (instead of in parallel), and each index would be loaded, searched, then unloaded again. Thanks you in advance! ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [August 1, 2019, 12:58pm UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/2 "2019-08-01T12:58:01Z")
Hi, thanks for sharing this here. Can you please explain your measurement approach in a little bit more detail? Specifically, I'd be interested in: 1. How did you measure shard size? 2. Are these indices [force-merged](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html)? Did you force-merge them to a single segment? 3. What do you mean by the metric "Used JVM Heap" that you present at the beginning of your post? 4. How did you measure JVM heap while running the query? 5. How many indices did you query? All of them or only a subset? The documentation talks about memory usage relative to regular indices so I suggest to base heap usage expectations not on absolute numbers but instead on comparisons to memory usage with regular indices vs. with frozen indices. Daniel ---
**Author:** ![Ali\_Reza\_Haghighat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_reza_haghighat/32/48590_2.png) [@Ali\_Reza\_Haghighat](https://discuss.elastic.co/u/Ali_Reza_Haghighat)\ **Post date:** [August 3, 2019, 1:46pm UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/3 "2019-08-03T13:46:33Z")
Hi Daniel, Thanks for your reply. 1. I have used `/_cat/indices` API to measure shard size. 2. No, I have not run the `_forcemerge`. 3. I wrote a python script to measure the used JVM heap by requesting `/_nodes/stats/jvm` and reading the "jvm-\>mem-\>heap\_used\_in\_bytes" field. 4. My script reads the JVM heap periodically (e.g., every 5 seconds). 5. I run `{"query" : {"match_all": {}}}` query on all indices. I will run some tests to compare the ratio of memory usage between the regular indices and frozen indices. But based on my old experiments, I think the Elasticsearch requires only 8 GB to handle a similar query on the same condition. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 3, 2019, 1:53pm UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/4 "2019-08-03T13:53:27Z")
You should forcemerge down to 1segment before you freeze indices. Try and see what difference it makes. ---
**Author:** ![Ali\_Reza\_Haghighat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_reza_haghighat/32/48590_2.png) [@Ali\_Reza\_Haghighat](https://discuss.elastic.co/u/Ali_Reza_Haghighat)\ **Post date:** [August 3, 2019, 2:02pm UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/5 "2019-08-03T14:02:40Z")
Thanks Christian, I tried to run force\_merge, but I wasn't successful. I run this command on an index with 1 shard and 23 segments: > curl -X POST '[http://localhost:9200/geonames2-0/\_forcemerge?max\_num\_segments=1](http://localhost:9200/geonames2-0/_forcemerge?max_num_segments=1)' I get the following result: > {"\_shards":{"total":1,"successful":1,"failed":0}} However, the index has still 23 segments (even after 2 hours). ---
**Author:** ![Ali\_Reza\_Haghighat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_reza_haghighat/32/48590_2.png) [@Ali\_Reza\_Haghighat](https://discuss.elastic.co/u/Ali_Reza_Haghighat)\ **Post date:** [August 3, 2019, 2:31pm UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/6 "2019-08-03T14:31:13Z")
Excuse me, it was my fault!!! I had run the `_forcemerge` query on a freezed index. After unfreezing the index, my `_forcemerge` request is now being processed. I can also see it using the `/_tasks` API. I will try to forcemerge all of my indices and then retry my experiment. BTW, it would be better if running a `_forcemerge` query on a freezed index returns a warning or error! ---
**Author:** ![Ali\_Reza\_Haghighat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_reza_haghighat/32/48590_2.png) [@Ali\_Reza\_Haghighat](https://discuss.elastic.co/u/Ali_Reza_Haghighat)\ **Post date:** [August 4, 2019, 12:49pm UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/7 "2019-08-04T12:49:51Z")
The `_forcemerge` operation decrease JVM heap usage of my indices dramatically (even when they are not freezed). [Here](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) we have a rule of thumb for computing required JVM heap based on the number of shards: > _A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap it has configured_ I just wonder whether there is a similar rule of thumb for forcemerged shards? In other words, I think for forcemerged indices we can have for example 50 shards per GB heap. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 4, 2019, 4:40pm UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/8 "2019-08-04T16:40:49Z")
It is cluster state size as well as heap usage that drives this recommendation, and it assumes best practices are being followed, which includes forcemerging indices that are no longer written to down to a single segment. ---
**Author:** ![Ali\_Reza\_Haghighat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_reza_haghighat/32/48590_2.png) [@Ali\_Reza\_Haghighat](https://discuss.elastic.co/u/Ali_Reza_Haghighat)\ **Post date:** [August 5, 2019, 2:17pm UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/9 "2019-08-05T14:17:22Z")
Thanks Christian! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 2, 2019, 2:17pm UTC](https://discuss.elastic.co/t/search-on-frozen-index-cause-huge-usage-of-jvm-heap/193101/10 "2019-09-02T14:17:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.