Indices Indexing Rate Max value
Indices Indexing Rate Mean value
Thread pool queue Max
Thread pool queue min
Thread pool rejected
As you can see, indexing rate is smooth more or less, but the thread queue size has spikes.
And based on the difference between Max and Mean values of queue size also, it is clear that queue size is varying highly.
I can see that these spikes are single data points (If I zoom in), It seems that queue size is zero most of the times, but suddenly spikes up for a small amount of time and goes back to zero.
I don't see any memory or cpu spikes,Disk IOPS spikes in kibana dashboards.
What are the possible reasons that this might be happening?
How do I know which metric is the bottle neck?
Is there a way to confirm if Disk IOPS is/isn't a bottle neck.
This ES isn't part of an ELK, so there isn't any bulk indexing happening.
My Cluster Configuration is:
3 m3.xlarge, 8GB Mem locked.