Full GC,Elasticsearch heap contains a lot of CompressingStoredFieldsReader

[psiitoy]

My cluster nodes always Full GC，I use MAT's function "Path to GC Roots" find the refrence is
IndicesQueryCache and CompressingStoredFieldsReader,

How to solve the question,IndicesQueryCache and CompressingStoredFieldsReader for what purpose ,why them in Thread named bulk.Thank you.

java verison:1.8.0_131
elasticsearch verison:2.1.2
lucene verison:5.4.1

As shown in figure :

[jpountz (Adrien Grand)]

This is a side-effect of having too many segments on a single node.

This means you need to reduce the number of shards per node (eg. by shrinking some indices) and/or reduce the number of segments per shard (by force-merging some indices, just beware to ONLY do this on READ-ONLY indices).

[psiitoy]

Thanks for your reply .what operation may cause this problem? Query or Index or when
just having too many segments auto?

[jpountz (Adrien Grand)]

It means too many segments are open at the same time, it is neither related to indexing or searching. You could close them by I presume you need them for indexing or searching so this is not really an option.

[psiitoy]

Our cluster has 28 index in 10 datanodes and divide into 400 shard,per index has 500 segments and 100GB docs.Can you give a parameter optimization suggestion for this case? Thank you.

[jpountz (Adrien Grand)]

Do you have 400 shards in total, per-index or per-node?
Also do you have 100GB of data per shard, per node, per index or in the entire cluster?

[psiitoy]

Hi Adrien, our cluster as follows : 400 shards in total(200 are replicas) and 100GB of data per node.

[jpountz (Adrien Grand)]

OK, this is consistent with the number of stored fields readers instances in the heap dump. How many processors do you have on each machine? Do you overwrite the size of thread pools or the processors setting?

[psiitoy]

80 core per machine.The thread pool setting is as follows json.

"threadpool": {
"search": {
"size": "128",
"queue": "1001"
},
"index": {
"size": "32",
"queue": "1000"
},
"bulk": {
"size": "32",
"queue": "1000"
}
}

[jpountz (Adrien Grand)]

Thank you. Things look sane to me, these 2GB of heap retained by stored fields readers are due to the fact that they keep some state per-thread per-segment. I don't think it makes sense to reduce the number of threads that you have, which is reasonable regarding your number of cores. So I think we should either keep things this way (2GB should not be that much for a beefy machine), or look into reducing the number of segments by force-merging read-only indices if any, or trying to have fewer shards overall in the cluster.

[psiitoy]

Thanks Adrien for quick reply! I will try your advice.