We are seeing performance problems with our elasticsearch cluster.
Our setup is running ES 6.6 with 6 nodes total where 2 are master and 4 are data. Each data node has 32GB memory and 16 available to the heap. We have one index spread over these 4 data nodes with a total of 10 primary shards and 10 replicas.
Total doc count is just below 1 billion at a size of 186GB (not counting replicas)
The heap consists of a high amount of field data, and cpu is always working with rebuilding global ordinals which can be seen in hot_threads api
We have a product in which we display a wordcloud, this wordcloud is currently built using a terms aggregation on an analyzed text field, meaning we are using fielddata:true on this field.
This text field contains peoples text answers on different questions given to them, meaning, the field contains sentences varying both in length and content.
Now, every guide, article or forum post I've read, strongly discourages this, due to the large amount of field data hogging the heap.
Therefore my initial plan was then to simply turn this field into a keyword field, and do the splitting of words myself on our side before I index them into elastic, basically doing what elastic does when analyzing a field, thus allowing us to not having to run fielddata:true.
However, the more I read about this, I get the feeling that this might not solve the underlying issues we are seeing with huge heaps and refreshes taking up to a minute (eager global ordinal rebuilds).
I'm getting the feeling the issue is more related to the high cardinality regardless of the field being declared as fielddata:true.
Am I correct in assuming that changing to a keyword field won't solve the actual issue, and if so, what are my options here?
Worth mentioning is that while the field is high cardinality, we never run this aggregation on the whole index but always on a sub set of documents, due to this I've thought about disabling eager global ordinals, thinking this will only build global ordinals for the relevant matching documents?