Elasticsearch high cpu usage

Elastic Stack Elasticsearch
#1

Hi everyone.

My Es cluster had high cpu usage at 3:30 am today, below is grafana monitoring:

image
image938×524 83.5 KB

image
image1883×602 290 KB

And the kibana monitoring:

image
image1838×750 189 KB

I don't have a clue, because in my nginx log files, I didn't see any request rise significantly at that time, and when I get up, es cpu usage had already dropped.

btw, in elastic cluster log files, no error logs were created at the time.

Do you have any other suggestions?

What should I do so that the next time I encounter a similar problem, I can better troubleshoot it.

Pretty thanks for any records.

#2

The request time for Search went way up at the same time, didn't it? Unusual search? Do you have Kibana users? Do you have the Kibana user that does big queries?

#3

What is the output from the _cluster/stats?pretty&human API?
What do the Elasticsearch logs show at that time?