Elasticsearch cluster performance degradation

Elastic Stack Elasticsearch
#1

We had a performance degradation for 5-10 minutes intervals. We looked into our monitoring tool and found there was high CPU utilization on the elasticsearch cluster. During this time we also noticed high fetch requests and cache eviction. What do the high fetch requests and high cache eviction implies?

Also, the rest of the metrics looked fine. Our slow query settings are set to 1 minute. So, we don't have all the queries which were executed during that time.

#2

Where are you seeing these high metrics?

#3

APM - SignalFX

#4

I'm not familiar with that or how it collects those.

If there is an increase in requests and cache evictions (assuming the cache it's measuring is the Elasticsearch one, and not OS level), then that might suggest a large-ish number of unique queries coming in in a burst.

Is there a corresponding increase in GC?

#5

There was increase in ES cache evictions. But, GC looked fine. I have not noticed the query requests going up during this time.

#6

So what's the high fetch requests metric?