I ran a _nodes/hot_threads and here is the output:
What should I do to diagnose further?
Hi,
It's a little hard to tell what is going on just from the thread dump. Looks like you have bulk indexing, searching, and some merging happening (at the time of that thread dump). What changed recently? Started indexing more data? # of shards increased? Added nodes?
Can you use something like SPM, point it to your ES cluster, and share charts from some of your other metrics? e.g. GC counts/timings, JVM memory pool utilization, query and index rates, merge chart, etc. That should give us more info about what's going on. http://sematext.com/spm to grab SPM.
Otis
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/