I have a classic ELK cluster for storing logs from my applications.
3 nodes 8Gb of RAM each box (6Gb dedicated to ES)
Running on AWS gp2 provisioned 3000 IOps hard drives.
version: 5.0.0 (5.2.2 gives the same results)
1 instance with default configuration.
Recently I realized that we can't catch up with the amount of data we are pushing into ES.
After x-pack installation I found out that I have a sawtooth pattern in the "Indexing rate" section.
It looks like the whole box suspends completely every 90 seconds.
I increased memory allocation from 4Gb to 6Gb which didn't help a lot.
The weird thing is that I can see the same behavior when I put ES off the load.
And these gaps are in all charts which look so weird.
I see two possible scenarios either ES box freezes periodically either X-Pack just messes monitoring packets.
Personally, it seems impossible to fall JVM heap to zero because of GC.
Off the load