# \[6.8.2\] Unusual Server Load **URL:** \ **Category:** Elasticsearch\ **Created:** [August 7, 2019, 6:08am UTC](https://discuss.elastic.co/t/6-8-2-unusual-server-load/194153 "2019-08-07T06:08:47Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [August 7, 2019, 6:08am UTC](https://discuss.elastic.co/t/6-8-2-unusual-server-load/194153/1 "2019-08-07T06:08:47Z")
Hi, I'm having trouble with dealing an odd pattern of load distribution on one of our clusters. I inadvertently posted in the wrong forum. Link is here: > [@\[6.8.2\] Very High Load on One Node](https://discuss.elastic.co/t/6-8-2-very-high-load-on-one-node/193973): > > Hi, I'm seeing an unusual load pattern on one of our logging clusters and am struggling to deal with it... Basically, I'm seeing a load average of 20+ on one of the data nodes. The other nodes are in their normal range. If I restart the node, the load moves to another node. I have tried shutting down all of the data nodes and masters, and restarting, with no change. Looking at hot\_threads I see search requests at the top of the list. However, I don't know if these are the actual cause of the … FYI we have 18 data nodes, 1250 shards and a limit of 3 index shards per node. So the loading pattern doesn't appear related to shard distribution. Thx D ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [August 7, 2019, 7:11am UTC](https://discuss.elastic.co/t/6-8-2-unusual-server-load/194153/2 "2019-08-07T07:11:05Z")
try using the [hot\_threads API](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/cluster-nodes-hot-threads.html) and paste the output here so we can check what java code is eating all the CPU. ---
**Author:** ![Denis\_Lamanov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/denis_lamanov/32/13111_2.png) [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)\ **Post date:** [August 7, 2019, 3:15pm UTC](https://discuss.elastic.co/t/6-8-2-unusual-server-load/194153/3 "2019-08-07T15:15:02Z")
Have the same situation since 6.8.0 ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [August 8, 2019, 8:08am UTC](https://discuss.elastic.co/t/6-8-2-unusual-server-load/194153/4 "2019-08-08T08:08:52Z")
@spinscale Hot Thread output for the node in question: Hot thread output can be found [here](https://pastebin.com/ypKafUZ8) Thx D ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [August 8, 2019, 9:43am UTC](https://discuss.elastic.co/t/6-8-2-unusual-server-load/194153/5 "2019-08-08T09:43:51Z")
are you doing a lot [sliced scroll](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/search-request-body.html#sliced-scroll) searches? The hot threads output seems to indicate that? Are you only hitting this one node with sliced scroll queries? ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [August 8, 2019, 3:23pm UTC](https://discuss.elastic.co/t/6-8-2-unusual-server-load/194153/6 "2019-08-08T15:23:05Z")
One of the devs may be. If so, I don't know who it is. Our data isn't indexed to be pinned to individual shards. How could this be happening? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [August 12, 2019, 4:14pm UTC](https://discuss.elastic.co/t/6-8-2-unusual-server-load/194153/7 "2019-08-12T16:14:56Z")
I think trying out to figure who is accessing your data sounds like a good idea. A scroll search is usually not issued by kibana (i.e. for a dashboard), so someone/some code needs to trigger this explicitely. As for Elasticsearch this is simply an open TCP connection, maybe using some more lowlevel tools could help, like tcpdump or netstat? This instance is not exposed to the internet I suppose? So you control who can access it? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 9, 2019, 4:15pm UTC](https://discuss.elastic.co/t/6-8-2-unusual-server-load/194153/8 "2019-09-09T16:15:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.