We have a huge elasticsearch cluster with 120 nodes and a lot of indices, One huge index around 650GB, many ~50GB weekly indices, some daily indices. They have different shard and replica settings. We run 1.7 version with 16GB heap size. Recently we are seeing lots of issues related to heavy GC and many nodes move out of the cluster and we have to restart the whole cluster sometimes. This is happening almost every day. A portion of documents that get indexed into the weekly indices and the large index are huge in size, with a few hundred fields. Some of the heap dumps show a considerable portion of heap being occupied by 'org.elasticsearch.index.mapper.MapperService' instances. We suspect this could be because of the field mapping which is definitely growing as the number of distinct fields in the documents for a type keep growing. Any suggestions or inputs to resolve this?
- Upgrade
- Review your mapping approach
- Upgrade
How large is your cluster state, how often does it change?
Thanks Mark,
The cluster state was actually huge, way more than (3x growth) what it used to be 3 weeks back. We figured out that a new type added recently which stores documents using a dynamic key for some nested fields, that has caused the increased mapping size for the new weekly indices. We will reindex them to a new index with marking dynamic mapping to false and check. Thanks for your inputs.
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.