Hi,
we have ES cluster v1.4.4 (x12 vcpu, 20GB RAM x4 nodes).
With time, CPU usage grows and the stays at top and search queues are shooting up.
After elasticsearch service restart, CPU usage drops down to ~40% and then gradually climbs up again.
on higher load, if service was just restarted - ES holds up at around 60% CPU usage, and then, with load drop CPU usage also drops. Now problem is, that when the load starts again - ES CPU usage climbs way higher than 60% - and then queries are starting to queue and response time increases dramatically.
How can we find, what is causing this CPU increase and isolate that problem?
./elasticsearch -v
Version: 1.4.4, Build: c88f77f/2015-02-19T13:05:36Z, JVM: 1.7.0_80
java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
_settings, jstack dump of one of data servers (with highest cpu load) and hot_threads are located in gist:
https://gist.github.com/edast/01d8be2b869f74559be5