[6.8.2] Unusual Server Load

Elastic Stack Elasticsearch
1

Hi,
I'm having trouble with dealing an odd pattern of load distribution on one of our clusters. I inadvertently posted in the wrong forum. Link is here:

FYI we have 18 data nodes, 1250 shards and a limit of 3 index shards per node. So the loading pattern doesn't appear related to shard distribution.

Thx
D

2

try using the hot_threads API and paste the output here so we can check what java code is eating all the CPU.

3

Have the same situation since 6.8.0

4

@spinscale Hot Thread output for the node in question:

Hot thread output can be found here

Thx
D

5

are you doing a lot sliced scroll searches? The hot threads output seems to indicate that? Are you only hitting this one node with sliced scroll queries?

6

One of the devs may be. If so, I don't know who it is. Our data isn't indexed to be pinned to individual shards. How could this be happening?

7

I think trying out to figure who is accessing your data sounds like a good idea. A scroll search is usually not issued by kibana (i.e. for a dashboard), so someone/some code needs to trigger this explicitely.

As for Elasticsearch this is simply an open TCP connection, maybe using some more lowlevel tools could help, like tcpdump or netstat?

This instance is not exposed to the internet I suppose? So you control who can access it?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.