# Single node with high cpu usage in large number of shards scenario

**URL:** <https://discuss.elastic.co/t/single-node-with-high-cpu-usage-in-large-number-of-shards-scenario/152743>\
**Category:** Elasticsearch\
**Created:** [October 17, 2018, 3:45am UTC](https://discuss.elastic.co/t/single-node-with-high-cpu-usage-in-large-number-of-shards-scenario/152743 "2018-10-17T03:45:34Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![howardhuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/howardhuang/32/69365_2.png) [@howardhuang](https://discuss.elastic.co/u/howardhuang)\
**Post date:** [October 17, 2018, 3:45am UTC](https://discuss.elastic.co/t/single-node-with-high-cpu-usage-in-large-number-of-shards-scenario/152743/1 "2018-10-17T03:45:34Z")

</div>

Hi All,

In my test environment, I setup a single ES 6.4.1 node, 1 core, 2GB memory (1GB heap).  
After creating 600 empty indices (without any data and read/write operation), each index has 20 shards, total 12000 shards, es process cpu goes to 90+% every several seconds.

I just want to figure out what cause high cpu usage, since it’s an empty data cluster and without any operation (read/write). I tried to use jstack, hot\_thread, but cann't find any clue, only see some active netty threads. Could anyone give some hint? Thanks a lot!

During testing, I disabled xpack features, and set some refresh, stats interval to a very long value to avoid impacting:

"cluster" : {  
"routing" : {  
"allocation" : {  
"node\_concurrent\_recoveries" : "100"  
}  
},  
"info" : {  
"update" : {  
"interval" : "1d"  
}  
}  
},  
"indices" : {  
"recovery" : {  
"max\_bytes\_per\_sec" : "400mb"  
}  
}

"settings" : {  
"index" : {  
"refresh\_interval" : "30m",  
"number\_of\_shards" : "20",  
"translog" : {  
"sync\_interval" : "1h",  
"durability" : "async"  
},  
"provided\_name" : "indexi0",  
"max\_result\_window" : "65536",  
"creation\_date" : "1539742381743",  
"store" : {  
"stats\_refresh\_interval" : "1h"  
},  
"number\_of\_replicas" : "0",  
"uuid" : "GxZ\_ADUzQo6mepQREJe5nA",  
"version" : {  
"created" : "6040199"  
}  
}  
}

I also modfied es code and disabled translog async and globle checkpoint scheduler. I cloud not see any frequent logs in trace level now. But cpu still goes high in every several seconds.

Thanks,  
Howard

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 17, 2018, 7:10am UTC](https://discuss.elastic.co/t/single-node-with-high-cpu-usage-in-large-number-of-shards-scenario/152743/2 "2018-10-17T07:10:17Z")

</div>

What was the reason you decided to go with thousands of shards?

This is way too many shards. Try to stick with as few shards as possible. Even one shard will be able to utilize your system resources.

---

<div class="post-metadata">

**Author:** ![howardhuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/howardhuang/32/69365_2.png) [@howardhuang](https://discuss.elastic.co/u/howardhuang)\
**Post date:** [October 17, 2018, 9:15am UTC](https://discuss.elastic.co/t/single-node-with-high-cpu-usage-in-large-number-of-shards-scenario/152743/3 "2018-10-17T09:15:36Z")

</div>

Hi Alex, thanks for the reply.

As in our product env, we have some small resource clusters with hundreds of shards scenario, we could see cpu usage very high every several seconds. I just scale out the shard number in test env and try to figure out the reason, try to find out which part of jobs used high cpu. But I cannot find any clue, since jstack, hot\_thread cannot supply any useful information.

So if I just create some indices with empty shard, no data read write, which would be the most costly part? Shard level statistic? Could you please guide me some information? Thanks a lot.

Thanks,

Howard

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 17, 2018, 10:46am UTC](https://discuss.elastic.co/t/single-node-with-high-cpu-usage-in-large-number-of-shards-scenario/152743/4 "2018-10-17T10:46:07Z")

</div>

each shard comes with its own requirements of memory/CPU/diskspace even when not a lot of data is indexed, thus you can just end up with memory pressure resulting in garbage collections, especially on smaller machines.

You should really try to reduce the number of shards, and definitaly read the [Designing for scale](https://www.elastic.co/guide/en/elasticsearch/guide/master/scale.html) in the definitive guide, this one in particular [https://www.elastic.co/guide/en/elasticsearch/guide/master/kagillion-shards.html](https://www.elastic.co/guide/en/elasticsearch/guide/master/kagillion-shards.html)

hope this helps!

--Alex

---

<div class="post-metadata">

**Author:** ![howardhuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/howardhuang/32/69365_2.png) [@howardhuang](https://discuss.elastic.co/u/howardhuang)\
**Post date:** [October 17, 2018, 2:04pm UTC](https://discuss.elastic.co/t/single-node-with-high-cpu-usage-in-large-number-of-shards-scenario/152743/5 "2018-10-17T14:04:35Z")

</div>

Thank you Alex! I finally found it's jvm gc cost. After using 1core 8GB memory(6GB heap), almost no cpu usage.

Thanks,  
Howard

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 14, 2018, 2:13pm UTC](https://discuss.elastic.co/t/single-node-with-high-cpu-usage-in-large-number-of-shards-scenario/152743/6 "2018-11-14T14:13:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.