# High CPU load on only one machine in a cluster of three **URL:** \ **Category:** Elasticsearch\ **Created:** [April 10, 2014, 9:37am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-one-machine-in-a-cluster-of-three/16918 "2014-04-10T09:37:15Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Huy\_Phan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huy_phan/32/1660_2.png) [@Huy\_Phan](https://discuss.elastic.co/u/Huy_Phan)\ **Post date:** [April 10, 2014, 9:37am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-one-machine-in-a-cluster-of-three/16918/1 "2014-04-10T09:37:15Z")
Hi guys, I think I'm having a similar issue as the last post in this thread: [https://groups.google.com/forum/#!searchin/elasticsearch/high$20load/elasticsearch/0EHITg5ndp4/sE-rsd4o-EEJ](https://groups.google.com/forum/#!searchin/elasticsearch/high%2420load/elasticsearch/0EHITg5ndp4/sE-rsd4o-EEJ) I'm running Elasticsearch 1.1.0 on 3 servers, each has 24 cores and 72GB memory, Elasticsearch is given 16GB for heap space. Out of the default settings, I got some tweaks in elasticsearch.yml: threadpool.bulk.type: fixed threadpool.bulk.queue\_size: 100 threadpool.index.type: fixed threadpool.index.queue\_size: 100 indices.memory.index\_buffer\_size: 30% indices.memory.min\_shard\_index\_buffer\_size: 12mb indices.memory.min\_index\_buffer\_size: 96mb I'm trying to index 150+ million records but when we reached 30 million documents, one of the server got really high CPU load (20-30) while the other two were not really busy (load was only 1-3). Also, there's only one index with 20 shards in the cluster, replication factor set to 1 and refresh\_interval is 30 seconds. Any suggestion to re-balance the load of this cluster ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ddf5e4eb-e903-49f9-9e1b-bd8063ac13cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ddf5e4eb-e903-49f9-9e1b-bd8063ac13cc%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Huy\_Phan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huy_phan/32/1660_2.png) [@Huy\_Phan](https://discuss.elastic.co/u/Huy_Phan)\ **Post date:** [April 10, 2014, 2:09pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-one-machine-in-a-cluster-of-three/16918/2 "2014-04-10T14:09:17Z")
Here's the stack trace of ES on the busy server:[stack trace on busy server · GitHub](https://gist.github.com/huyphan/e543119a65861ab89e54) And here's on another low CPU load server: [stack trace on low CPU load server · GitHub](https://gist.github.com/huyphan/ee0c1a3bcd48f5b8ba28) On Thursday, 10 April 2014 17:37:15 UTC+8, Huy Phan wrote: > Hi guys, > > I think I'm having a similar issue as the last post in this thread: > [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/high$20load/elasticsearch/0EHITg5ndp4/sE-rsd4o-EEJ) > > I'm running Elasticsearch 1.1.0 on 3 servers, each has 24 cores and 72GB > memory, Elasticsearch is given 16GB for heap space. Out of the default > settings, I got some tweaks in elasticsearch.yml: > > threadpool.bulk.type: fixed > threadpool.bulk.queue\_size: 100 > > threadpool.index.type: fixed > threadpool.index.queue\_size: 100 > > indices.memory.index\_buffer\_size: 30% > indices.memory.min\_shard\_index\_buffer\_size: 12mb > indices.memory.min\_index\_buffer\_size: 96mb > > I'm trying to index 150+ million records but when we reached 30 million > documents, one of the server got really high CPU load (20-30) while the > other two were not really busy (load was only 1-3). > > Also, there's only one index with 20 shards in the cluster, replication > factor set to 1 and refresh\_interval is 30 seconds. > > Any suggestion to re-balance the load of this cluster ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53385783-53fd-4f09-8904-ae321f33ffa4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/53385783-53fd-4f09-8904-ae321f33ffa4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Huy\_Phan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huy_phan/32/1660_2.png) [@Huy\_Phan](https://discuss.elastic.co/u/Huy_Phan)\ **Post date:** [April 11, 2014, 3:57am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-one-machine-in-a-cluster-of-three/16918/3 "2014-04-11T03:57:50Z")
Hi, I think I know the reason of this issue, it looks that the busy server is handling most of the Bulk request: While other nodes rarely share the load: I supposed that Bulk requests should be distributed evenly among the servers but turns out that they are not. Any thoughts on this? Thanks, Huy Phan On Thursday, 10 April 2014 22:09:17 UTC+8, Huy Phan wrote: > Here's the stack trace of ES on the busy server:[stack trace on busy server · GitHub](https://gist.github.com/huyphan/e543119a65861ab89e54) > And here's on another low CPU load server: > [stack trace on low CPU load server · GitHub](https://gist.github.com/huyphan/ee0c1a3bcd48f5b8ba28) > > On Thursday, 10 April 2014 17:37:15 UTC+8, Huy Phan wrote: > > > Hi guys, > > > > I think I'm having a similar issue as the last post in this thread: > > [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/high$20load/elasticsearch/0EHITg5ndp4/sE-rsd4o-EEJ) > > > > I'm running Elasticsearch 1.1.0 on 3 servers, each has 24 cores and 72GB > > memory, Elasticsearch is given 16GB for heap space. Out of the default > > settings, I got some tweaks in elasticsearch.yml: > > > > threadpool.bulk.type: fixed > > threadpool.bulk.queue\_size: 100 > > > > threadpool.index.type: fixed > > threadpool.index.queue\_size: 100 > > > > indices.memory.index\_buffer\_size: 30% > > indices.memory.min\_shard\_index\_buffer\_size: 12mb > > indices.memory.min\_index\_buffer\_size: 96mb > > > > I'm trying to index 150+ million records but when we reached 30 million > > documents, one of the server got really high CPU load (20-30) while the > > other two were not really busy (load was only 1-3). > > > > Also, there's only one index with 20 shards in the cluster, replication > > factor set to 1 and refresh\_interval is 30 seconds. > > > > Any suggestion to re-balance the load of this cluster ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7368b631-eda2-49ee-9df7-a574148c117f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7368b631-eda2-49ee-9df7-a574148c117f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Huy\_Phan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huy_phan/32/1660_2.png) [@Huy\_Phan](https://discuss.elastic.co/u/Huy_Phan)\ **Post date:** [April 11, 2014, 4:00am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-one-machine-in-a-cluster-of-three/16918/4 "2014-04-11T04:00:52Z")
Hi, I think I know the reason of this issue, it looks that the busy server is handling most of the Bulk request while other nodes rarely share the load (screenshot attached) I supposed that Bulk requests should be distributed evenly among the servers but turns out that they are not. Any thoughts on this? Thanks, Huy Phan On Thursday, 10 April 2014 22:09:17 UTC+8, Huy Phan wrote: > Here's the stack trace of ES on the busy server:[stack trace on busy server · GitHub](https://gist.github.com/huyphan/e543119a65861ab89e54) > And here's on another low CPU load server: > [stack trace on low CPU load server · GitHub](https://gist.github.com/huyphan/ee0c1a3bcd48f5b8ba28) > > On Thursday, 10 April 2014 17:37:15 UTC+8, Huy Phan wrote: > > > Hi guys, > > > > I think I'm having a similar issue as the last post in this thread: > > [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/high$20load/elasticsearch/0EHITg5ndp4/sE-rsd4o-EEJ) > > > > I'm running Elasticsearch 1.1.0 on 3 servers, each has 24 cores and 72GB > > memory, Elasticsearch is given 16GB for heap space. Out of the default > > settings, I got some tweaks in elasticsearch.yml: > > > > threadpool.bulk.type: fixed > > threadpool.bulk.queue\_size: 100 > > > > threadpool.index.type: fixed > > threadpool.index.queue\_size: 100 > > > > indices.memory.index\_buffer\_size: 30% > > indices.memory.min\_shard\_index\_buffer\_size: 12mb > > indices.memory.min\_index\_buffer\_size: 96mb > > > > I'm trying to index 150+ million records but when we reached 30 million > > documents, one of the server got really high CPU load (20-30) while the > > other two were not really busy (load was only 1-3). > > > > Also, there's only one index with 20 shards in the cluster, replication > > factor set to 1 and refresh\_interval is 30 seconds. > > > > Any suggestion to re-balance the load of this cluster ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3bb512b4-1874-4daa-9e10-d8f48477387d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3bb512b4-1874-4daa-9e10-d8f48477387d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:36am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-one-machine-in-a-cluster-of-three/16918/5 "2017-07-06T01:36:37Z")