# Full GC,Elasticsearch heap contains a lot of CompressingStoredFieldsReader **URL:** \ **Category:** Elasticsearch\ **Created:** [August 10, 2017, 2:28am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538 "2017-08-10T02:28:42Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![psiitoy](https://avatars.discourse-cdn.com/v4/letter/p/fbc32d/32.png) [@psiitoy](https://discuss.elastic.co/u/psiitoy)\ **Post date:** [August 10, 2017, 2:28am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/1 "2017-08-10T02:28:42Z")
My cluster nodes always Full GC，I use MAT's function "Path to GC Roots" find the refrence is IndicesQueryCache and CompressingStoredFieldsReader, How to solve the question,IndicesQueryCache and CompressingStoredFieldsReader for what purpose ,why them in Thread named bulk.Thank you. java verison:1.8.0\_131 elasticsearch verison:2.1.2 lucene verison:5.4.1 As shown in figure : ![239mat2](https://us1.discourse-cdn.com/elastic/original/3X/0/4/04e0975aacb150b89e647a8562e90ddae1911416.png) ![239mat6](https://us1.discourse-cdn.com/elastic/original/3X/2/9/293bf85496618feb65eb5d64f5ea1d79c8ce92fb.png) ![239mat5](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a909f08db0c95004e332ab57bfa7b684785f5c57.png) ![239mat4](https://us1.discourse-cdn.com/elastic/original/3X/7/3/73b1c8c8e0a46668e1355488c0fc743287c4b8d9.png) ![239mat3](https://us1.discourse-cdn.com/elastic/original/3X/8/5/85845a151763c34bf8d5415c176aba117e996101.png) ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 11, 2017, 8:15am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/2 "2017-08-11T08:15:51Z")
This is a side-effect of having too many segments on a single node. This means you need to reduce the number of shards per node (eg. by shrinking some indices) and/or reduce the number of segments per shard (by force-merging some indices, just beware to ONLY do this on READ-ONLY indices). ---
**Author:** ![psiitoy](https://avatars.discourse-cdn.com/v4/letter/p/fbc32d/32.png) [@psiitoy](https://discuss.elastic.co/u/psiitoy)\ **Post date:** [August 11, 2017, 12:01pm UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/3 "2017-08-11T12:01:50Z")
Thanks for your reply .what operation may cause this problem? Query or Index or when just having too many segments auto? ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 11, 2017, 12:23pm UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/4 "2017-08-11T12:23:17Z")
It means too many segments are open at the same time, it is neither related to indexing or searching. You could close them by I presume you need them for indexing or searching so this is not really an option. ---
**Author:** ![psiitoy](https://avatars.discourse-cdn.com/v4/letter/p/fbc32d/32.png) [@psiitoy](https://discuss.elastic.co/u/psiitoy)\ **Post date:** [August 14, 2017, 2:54am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/5 "2017-08-14T02:54:57Z")
Our cluster has 28 index in 10 datanodes and divide into 400 shard,per index has 500 segments and 100GB docs.Can you give a parameter optimization suggestion for this case? Thank you. ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 14, 2017, 7:11am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/6 "2017-08-14T07:11:12Z")
Do you have 400 shards in total, per-index or per-node? Also do you have 100GB of data per shard, per node, per index or in the entire cluster? ---
**Author:** ![psiitoy](https://avatars.discourse-cdn.com/v4/letter/p/fbc32d/32.png) [@psiitoy](https://discuss.elastic.co/u/psiitoy)\ **Post date:** [August 14, 2017, 8:28am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/7 "2017-08-14T08:28:57Z")
Hi Adrien, our cluster as follows : 400 shards in total(200 are replicas) and 100GB of data per node. ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 14, 2017, 8:56am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/8 "2017-08-14T08:56:52Z")
OK, this is consistent with the number of stored fields readers instances in the heap dump. How many processors do you have on each machine? Do you overwrite the size of thread pools or the `processors` setting? ---
**Author:** ![psiitoy](https://avatars.discourse-cdn.com/v4/letter/p/fbc32d/32.png) [@psiitoy](https://discuss.elastic.co/u/psiitoy)\ **Post date:** [August 14, 2017, 9:18am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/9 "2017-08-14T09:18:12Z")
80 core per machine.The thread pool setting is as follows json. "threadpool": { "search": { "size": "128", "queue": "1001" }, "index": { "size": "32", "queue": "1000" }, "bulk": { "size": "32", "queue": "1000" } } ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 14, 2017, 9:40am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/10 "2017-08-14T09:40:56Z")
Thank you. Things look sane to me, these 2GB of heap retained by stored fields readers are due to the fact that they keep some state per-thread per-segment. I don't think it makes sense to reduce the number of threads that you have, which is reasonable regarding your number of cores. So I think we should either keep things this way (2GB should not be that much for a beefy machine), or look into reducing the number of segments by force-merging read-only indices if any, or trying to have fewer shards overall in the cluster. ---
**Author:** ![psiitoy](https://avatars.discourse-cdn.com/v4/letter/p/fbc32d/32.png) [@psiitoy](https://discuss.elastic.co/u/psiitoy)\ **Post date:** [August 14, 2017, 9:53am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/11 "2017-08-14T09:53:23Z")
Thanks Adrien for quick reply! I will try your advice. 🤗 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 11, 2017, 9:53am UTC](https://discuss.elastic.co/t/full-gc-elasticsearch-heap-contains-a-lot-of-compressingstoredfieldsreader/96538/12 "2017-09-11T09:53:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.