QQ: what would cause IndexWriter memory to constantly spike?
I'm indexing data to a new 3 node ES6.1 cluster running on i3.xlarge instances. Each node is showing roughly equal numbers for index request rate, disk usage, doc counts, shard counts, and merge activity. I'm indexing almost exclusively to two indices, each with 3 shards evenly distributed to the 3 nodes. But one node is GCing a lot and the system load is 4-5x the others. I've restarted the node and quadrupled the heap from 4GB to 16GB, but the frequent GCs are still happening. There are two differences between the problem node and the two well-bahaved nodes:
- In Monitoring->Nodes->Node 2->Advanced->Index Memory (Elasticsearch), there are frequent Index Writer memory spikes to 20MB. The baseline of ~1MB is what the other two nodes hover at all the time, without any spikes. My hunch is that this is what's creating work for the GC.
iotopshows the problem node writing about 3-4x more to disk than the other two nodes.
My best guess at this point is that the node is trying to merge segments and failing, but I'm not seeing any reports of that in the logs. Can anyone suggest what else I should look at, or what transient logging I should enable to shine a light on what is causing that IndexWriter memory to be constantly allocated?
Below are snapshots from Monitoring showing some of the differences, including the Index Writer spikes.