zxx_bd
March 18, 2021, 7:26am
1
Hi, guys, we are running an ES cluster with 7.1.1 in production, and everything looks good while one day, a data node's CPU jumps to 90%+ suddenly, and all of the queries related to the index on that nodes were time out. However, when we restarted the ES process on that machine, everything goes back to normal and the problem could not be reproduced again. We have faced twice of this problem on the same cluster and we have no idea when will this happen again. Could anyone tell me how could I debug this problem?
Here is some information related to the ES cluster:
The flame graph on that node when the CPU was high.
It seems that it is not related to any special queries. When this happened, any query could cause the CPU jumps to 90%+, and when restarted the ES process, we replayed the same queries and they didn't cause the phenomenon again.
Thanks for any help!
warkolm
March 22, 2021, 1:46am
2
What do hot threads and slow log show for that node?
zxx_bd
March 23, 2021, 4:15am
3
hello, @warkolm thanks for your replying. I didn't record the hot threads and slow log at that time. I will provide this information for you later if it happened again.
warkolm
March 23, 2021, 11:34pm
4
Head into this particular node's monitoring page and see if there is anything else that correlates with the CPU spike.
zxx_bd
April 8, 2021, 5:46am
5
Hi,
@warkolm , the phenomenon happened again, and here is the image of the node's monitoring page of that node.
And here is the part of the hot_thread of that node:
{xpack.installed=true}
Hot threads at 2021-04-08T05:20:44.153Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
87.9% (439.7ms out of 500ms) cpu usage by thread 'elasticsearch[node-1][search][T#13]'
76.7% (383.7ms out of 500ms) cpu usage by thread 'elasticsearch[node-1][search][T#10]'
warkolm
April 8, 2021, 5:57am
6
Looks like a pretty heavy search based on that.
zxx_bd
April 8, 2021, 6:13am
7
But the problem is that, when I remove or restart this node, and use the exactly same queries to that cluster and everything works fine. The index and query are still the same.
system
May 6, 2021, 6:13am
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
