# 1 of 10 nodes CPU bound **URL:** \ **Category:** Elasticsearch\ **Created:** [December 19, 2016, 5:08pm UTC](https://discuss.elastic.co/t/1-of-10-nodes-cpu-bound/69476 "2016-12-19T17:08:58Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jazz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jazz/32/6943_2.png) [@jazz](https://discuss.elastic.co/u/jazz)\ **Post date:** [December 19, 2016, 5:08pm UTC](https://discuss.elastic.co/t/1-of-10-nodes-cpu-bound/69476/1 "2016-12-19T17:08:58Z")
We have a cluster with 10 machines with 10 data nodes, 3 master nodes and 10 replicated shards. Sometimes, we run many percolations and searches, and the CPU usages raises on all nodes, but not evenly. 1 node rises to 100% CPU usages while all the others rises to 50-80% of CPU usage. This makes most searches really slow (~20-25s). - Can I determine why this node takes 100% CPU time but not the others? - Can I do something to distribute the load more evenly? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [December 19, 2016, 5:23pm UTC](https://discuss.elastic.co/t/1-of-10-nodes-cpu-bound/69476/2 "2016-12-19T17:23:21Z")
Do you use custom routing of docs? This may account for uneven-ness. > [@jazz](#): > > Sometimes, we run many percolations This stands out as a possible culprit. If you have a particularly nasty percolator query that might be the cause of the imbalance. The best way to start is to use the hot threads API to see what's going on while under CPU pressure: > **[Nodes hot\_threads | Elasticsearch Guide \[5.1\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/cluster-nodes-hot-threads.html)** ---
**Author:** ![jazz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jazz/32/6943_2.png) [@jazz](https://discuss.elastic.co/u/jazz)\ **Post date:** [December 19, 2016, 5:37pm UTC](https://discuss.elastic.co/t/1-of-10-nodes-cpu-bound/69476/3 "2016-12-19T17:37:50Z")
> [@Mark\_Harwood](#): > > Do you use custom routing of docs? This may account for uneven-ness. No. > [@Mark\_Harwood](#): > > This stands out as a possible culprit. If you have a particularly nasty percolator query that might be the cause of the imbalance.The best way to start is to use the hot threads API to see what's going on while under CPU pressure:[Nodes hot\_threads | Elasticsearch Guide [5.1] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/cluster-nodes-hot-threads.html) It fluctuates highly. Most of the time it shows only 3 hot threads (on a hardware with 4 hyper-threaded CPUs with ES taking 800% of CPU time). Shouldn't it show me 8 hot threads ? ---
**Author:** ![jazz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jazz/32/6943_2.png) [@jazz](https://discuss.elastic.co/u/jazz)\ **Post date:** [December 19, 2016, 6:03pm UTC](https://discuss.elastic.co/t/1-of-10-nodes-cpu-bound/69476/4 "2016-12-19T18:03:27Z")
> [@jazz](#): > > Shouldn't it show me 8 hot threads ? OK, I read the doc... > threads: number of hot threads to provide, defaults to 3. ---
**Author:** ![jazz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jazz/32/6943_2.png) [@jazz](https://discuss.elastic.co/u/jazz)\ **Post date:** [December 20, 2016, 1:16pm UTC](https://discuss.elastic.co/t/1-of-10-nodes-cpu-bound/69476/5 "2016-12-20T13:16:43Z")
Now I can see that all threads are doing searches. How can I find from the call stacks what is expensive in the searches? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 17, 2017, 1:17pm UTC](https://discuss.elastic.co/t/1-of-10-nodes-cpu-bound/69476/6 "2017-01-17T13:17:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.