# High CPU Load on only some of the machines in a cluster

**URL:** <https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751>\
**Category:** Elasticsearch\
**Created:** [February 20, 2012, 4:01am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751 "2012-02-20T04:01:57Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chris\_Rode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_rode/32/2923_2.png) [@Chris\_Rode](https://discuss.elastic.co/u/Chris_Rode)\
**Post date:** [February 20, 2012, 4:01am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/1 "2012-02-20T04:01:57Z")

</div>

Hi all

I have not been able to find anything to explain why this may be occurring..

We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10  
minutes. When we put the cluster under load though, only 3 of the 7 nodes  
get any load on them, and that quickly builds up to a point where queries  
are timing out.

We originally had a replication factor of 3 set, and have just updated it  
to 6 (shard count is 20 per index), however this does not seem to have done  
the trick

Any ideas?

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 20, 2012, 1:12pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/2 "2012-02-20T13:12:14Z")

</div>

Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?

On Monday, February 20, 2012 at 6:01 AM, Chris Rode wrote:

> Hi all
> 
> I have not been able to find anything to explain why this may be occurring..
> 
> We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> 
> We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> 
> Any ideas?

---

<div class="post-metadata">

**Author:** ![Chris\_Rode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_rode/32/2923_2.png) [@Chris\_Rode](https://discuss.elastic.co/u/Chris_Rode)\
**Post date:** [February 21, 2012, 6:16am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/3 "2012-02-21T06:16:11Z")

</div>

Hey Shay

Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
head I can see that each index is not spread evenly. Trying to use  
index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
now...

Anything else I should try?

On Feb 21, 12:12 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> 
> On Monday, February 20, 2012 at 6:01 AM, Chris Rode wrote:
> 
> > Hi all
> 
> > I have not been able to find anything to explain why this may be occurring..
> 
> > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> 
> > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> 
> > Any ideas?

---

<div class="post-metadata">

**Author:** ![Chris\_Rode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_rode/32/2923_2.png) [@Chris\_Rode](https://discuss.elastic.co/u/Chris_Rode)\
**Post date:** [February 21, 2012, 11:32pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/4 "2012-02-21T23:32:54Z")

</div>

UPDATE: Unfortunately applying this transiently baulked.. however I  
have applied manually to each node in the cluster. After the dust  
settled, the shards still aren't distributed fairly.  
We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
= 12 so this should be fine) and some nodes still have 20 shards on an  
index.

We are using 18.6 is this an 18.7 only setting by any chance?

On Feb 21, 5:16 pm, Chris Rode [cir...@gmail.com](mailto:cir...@gmail.com) wrote:

> Hey Shay
> 
> Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
> head I can see that each index is not spread evenly. Trying to use  
> index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
> now...
> 
> Anything else I should try?
> 
> On Feb 21, 12:12 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> 
> > On Monday, February 20, 2012 at 6:01 AM, Chris Rode wrote:
> > 
> > > Hi all
> 
> > > I have not been able to find anything to explain why this may be occurring..
> 
> > > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> 
> > > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> 
> > > Any ideas?

---

<div class="post-metadata">

**Author:** ![jacque74](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jacque74](https://discuss.elastic.co/u/jacque74)\
**Post date:** [February 22, 2012, 3:08am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/5 "2012-02-22T03:08:40Z")

</div>

The thing to do is to do a jstack on the alas. search process. (Use  
paste bin to report link here). Do it during load. We will be able to  
tell which is happening. So far I noted at least few major overload  
conditions. 1). When index is cold and you are doing bulk inserts,  
you may get bloom filter load to kill your disk IO? (Shay, is there  
way to pace this somehow, the IOwait and waiting threads go to  
500-1000). 2). You are doing massive merges or flushes, potentially  
your refresh is not set to -1 when you are indexing. 3). Your disks  
are too slow, check iostat -xdm 1 report for 30 seconds. 4). You are  
running heap memory. (check log for oome msgs).

-Jack

On Feb 21, 3:32 pm, Chris Rode [cir...@gmail.com](mailto:cir...@gmail.com) wrote:

> UPDATE: Unfortunately applying this transiently baulked.. however I  
> have applied manually to each node in the cluster. After the dust  
> settled, the shards still aren't distributed fairly.  
> We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
> = 12 so this should be fine) and some nodes still have 20 shards on an  
> index.
> 
> We are using 18.6 is this an 18.7 only setting by any chance?
> 
> On Feb 21, 5:16 pm, Chris Rode [cir...@gmail.com](mailto:cir...@gmail.com) wrote:
> 
> > Hey Shay
> 
> > Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
> > head I can see that each index is not spread evenly. Trying to use  
> > index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
> > now...
> 
> > Anything else I should try?
> 
> > On Feb 21, 12:12 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > > Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> 
> > > On Monday, February 20, 2012 at 6:01 AM, Chris Rode wrote:
> > > 
> > > > Hi all
> 
> > > > I have not been able to find anything to explain why this may be occurring..
> 
> > > > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> 
> > > > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> 
> > > > Any ideas?

---

<div class="post-metadata">

**Author:** ![Chris\_Rode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_rode/32/2923_2.png) [@Chris\_Rode](https://discuss.elastic.co/u/Chris_Rode)\
**Post date:** [February 24, 2012, 4:29am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/6 "2012-02-24T04:29:08Z")

</div>

Thanks for your offer to help jack

[http://pastebin.com/ZFHJzX9Y](http://pastebin.com/ZFHJzX9Y)

That is the Jstack trace you asked for. We have 4 nodes now on  
m1.xlarge with an mdadm raid 0 across the 4 drives.

We are doing indexing whilst we run the load as well as straight  
queries. We havnt turned off the refresh but have have turned it down  
to 10seconds

gateway.index\_refresh: 10

I have also skitched a copy of the cluster allocation tables in  
elasticsearch-head

> **[Skitch | Evernote](https://evernote.com/products/skitch)**
>
> Get your point across with fewer words using annotation, shapes and sketches, so that your ideas become reality faster.

Any help you can give would be much appreciated

Cheers

On Feb 22, 2:08 pm, Jack Levin [magn...@gmail.com](mailto:magn...@gmail.com) wrote:

> The thing to do is to do a jstack on the alas. search process. (Use  
> paste bin to report link here). Do it during load. We will be able to  
> tell which is happening. So far I noted at least few major overload  
> conditions. 1). When index is cold and you are doing bulk inserts,  
> you may get bloom filter load to kill your disk IO? (Shay, is there  
> way to pace this somehow, the IOwait and waiting threads go to  
> 500-1000). 2). You are doing massive merges or flushes, potentially  
> your refresh is not set to -1 when you are indexing. 3). Your disks  
> are too slow, check iostat -xdm 1 report for 30 seconds. 4). You are  
> running heap memory. (check log for oome msgs).
> 
> -Jack
> 
> On Feb 21, 3:32 pm,ChrisRode[cir...@gmail.com](mailto:cir...@gmail.com) wrote:
> 
> > UPDATE: Unfortunately applying this transiently baulked.. however I  
> > have applied manually to each node in the cluster. After the dust  
> > settled, the shards still aren't distributed fairly.  
> > We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
> > = 12 so this should be fine) and some nodes still have 20 shards on an  
> > index.
> 
> > We are using 18.6 is this an 18.7 only setting by any chance?
> 
> > On Feb 21, 5:16 pm,ChrisRode[cir...@gmail.com](mailto:cir...@gmail.com) wrote:
> 
> > > Hey Shay
> 
> > > Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
> > > head I can see that each index is not spread evenly. Trying to use  
> > > index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
> > > now...
> 
> > > Anything else I should try?
> 
> > > On Feb 21, 12:12 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > > > Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> 
> > > > On Monday, February 20, 2012 at 6:01 AM,ChrisRodewrote:
> > > > 
> > > > > Hi all
> 
> > > > > I have not been able to find anything to explain why this may be occurring..
> 
> > > > > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> 
> > > > > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> 
> > > > > Any ideas?

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 26, 2012, 7:22pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/7 "2012-02-26T19:22:37Z")

</div>

The setting only applies in 0.19, but you did not answer the question with a bit more detailed information (how many indices, how many shards, are they not evenly balanced)?

On Wednesday, February 22, 2012 at 1:32 AM, Chris Rode wrote:

> UPDATE: Unfortunately applying this transiently baulked.. however I  
> have applied manually to each node in the cluster. After the dust  
> settled, the shards still aren't distributed fairly.  
> We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
> = 12 so this should be fine) and some nodes still have 20 shards on an  
> index.
> 
> We are using 18.6 is this an 18.7 only setting by any chance?
> 
> On Feb 21, 5:16 pm, Chris Rode \<[cir...@gmail.com](mailto:cir...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
> 
> > Hey Shay
> > 
> > Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
> > head I can see that each index is not spread evenly. Trying to use  
> > index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
> > now...
> > 
> > Anything else I should try?
> > 
> > On Feb 21, 12:12 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
> > 
> > > Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> > 
> > > On Monday, February 20, 2012 at 6:01 AM, Chris Rode wrote:
> > > 
> > > > Hi all
> > 
> > > > I have not been able to find anything to explain why this may be occurring..
> > 
> > > > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> > 
> > > > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> > 
> > > > Any ideas?

---

<div class="post-metadata">

**Author:** ![Chris\_Rode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_rode/32/2923_2.png) [@Chris\_Rode](https://discuss.elastic.co/u/Chris_Rode)\
**Post date:** [February 26, 2012, 11:25pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/8 "2012-02-26T23:25:16Z")

</div>

Sorry Shay

The image attached via skitch shows the cluster and distribution at  
the time of the Jstack trace

We have 4 nodes, with 3 indexes and a replication factor of 3. Each  
index has 20 shards. These are running across two availability zones  
on amazon.

When we turn down the replication factor the nodes get an even number  
of shards, however they do not get an even number of shards per index.  
This means that indexes with higher load do not utilise the entire  
clusters resources

We are currently using 0.18.7

On Feb 27, 6:22 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> The setting only applies in 0.19, but you did not answer the question with a bit more detailed information (how many indices, how many shards, are they not evenly balanced)?
> 
> On Wednesday, February 22, 2012 at 1:32 AM, Chris Rode wrote:
> 
> > UPDATE: Unfortunately applying this transiently baulked.. however I  
> > have applied manually to each node in the cluster. After the dust  
> > settled, the shards still aren't distributed fairly.  
> > We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
> > = 12 so this should be fine) and some nodes still have 20 shards on an  
> > index.
> 
> > We are using 18.6 is this an 18.7 only setting by any chance?
> 
> > On Feb 21, 5:16 pm, Chris Rode \<[cir...@gmail.com](mailto:cir...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
> > 
> > > Hey Shay
> 
> > > Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
> > > head I can see that each index is not spread evenly. Trying to use  
> > > index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
> > > now...
> 
> > > Anything else I should try?
> 
> > > On Feb 21, 12:12 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
> 
> > > > Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> 
> > > > On Monday, February 20, 2012 at 6:01 AM, Chris Rode wrote:
> > > > 
> > > > > Hi all
> 
> > > > > I have not been able to find anything to explain why this may be occurring..
> 
> > > > > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> 
> > > > > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> 
> > > > > Any ideas?

---

<div class="post-metadata">

**Author:** ![Chris\_Rode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_rode/32/2923_2.png) [@Chris\_Rode](https://discuss.elastic.co/u/Chris_Rode)\
**Post date:** [February 27, 2012, 12:36am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/9 "2012-02-27T00:36:48Z")

</div>

This new skitch illustrates what I was saying about the index-node  
balancing behavior

> **[Skitch | Evernote](https://evernote.com/products/skitch)**
>
> Get your point across with fewer words using annotation, shapes and sketches, so that your ideas become reality faster.

On Feb 27, 10:25 am, Chris Rode [cir...@gmail.com](mailto:cir...@gmail.com) wrote:

> Sorry Shay
> 
> The image attached via skitch shows the cluster and distribution at  
> the time of the Jstack trace
> 
> We have 4 nodes, with 3 indexes and a replication factor of 3. Each  
> index has 20 shards. These are running across two availability zones  
> on amazon.
> 
> When we turn down the replication factor the nodes get an even number  
> of shards, however they do not get an even number of shards per index.  
> This means that indexes with higher load do not utilise the entire  
> clusters resources
> 
> We are currently using 0.18.7
> 
> On Feb 27, 6:22 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > The setting only applies in 0.19, but you did not answer the question with a bit more detailed information (how many indices, how many shards, are they not evenly balanced)?
> 
> > On Wednesday, February 22, 2012 at 1:32 AM,ChrisRodewrote:
> > 
> > > UPDATE: Unfortunately applying this transiently baulked.. however I  
> > > have applied manually to each node in the cluster. After the dust  
> > > settled, the shards still aren't distributed fairly.  
> > > We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
> > > = 12 so this should be fine) and some nodes still have 20 shards on an  
> > > index.
> 
> > > We are using 18.6 is this an 18.7 only setting by any chance?
> 
> > > On Feb 21, 5:16 pm,ChrisRode\<[cir...@gmail.com](mailto:cir...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
> > > 
> > > > Hey Shay
> 
> > > > Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
> > > > head I can see that each index is not spread evenly. Trying to use  
> > > > index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
> > > > now...
> 
> > > > Anything else I should try?
> 
> > > > On Feb 21, 12:12 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
> 
> > > > > Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> 
> > > > > On Monday, February 20, 2012 at 6:01 AM,ChrisRodewrote:
> > > > > 
> > > > > > Hi all
> 
> > > > > > I have not been able to find anything to explain why this may be occurring..
> 
> > > > > > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> 
> > > > > > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> 
> > > > > > Any ideas?

---

<div class="post-metadata">

**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\
**Post date:** [February 27, 2012, 3:12am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/10 "2012-02-27T03:12:19Z")

</div>

Chris,

ES does not distribute shards to accomplish " even number of shards per  
index". It distributes to have even number of shards per node in total.  
Currently in ES, when you have indices with drastically different sizes,  
you end up with unbalanced load.  
Why do you have a separate index with 20 shards for the index with 106  
documents, is it supposed to grow? The easiest way to resolve the problem  
is to have only a single index with different types for 2nd and 3rd index,  
rather than having separate indices.

Regards,  
Berkay Mollamustafaoglu  
mberkay on yahoo, google and skype

On Sun, Feb 26, 2012 at 7:36 PM, Chris Rode [cirode@gmail.com](mailto:cirode@gmail.com) wrote:

> This new skitch illustrates what I was saying about the index-node  
> balancing behavior
> 
> [Skitch | Evernote](https://skitch.com/christopherrode/8r9xr/elasticsearch-head)
> 
> On Feb 27, 10:25 am, Chris Rode [cir...@gmail.com](mailto:cir...@gmail.com) wrote:
> 
> > Sorry Shay
> > 
> > The image attached via skitch shows the cluster and distribution at  
> > the time of the Jstack trace
> > 
> > We have 4 nodes, with 3 indexes and a replication factor of 3. Each  
> > index has 20 shards. These are running across two availability zones  
> > on amazon.
> > 
> > When we turn down the replication factor the nodes get an even number  
> > of shards, however they do not get an even number of shards per index.  
> > This means that indexes with higher load do not utilise the entire  
> > clusters resources
> > 
> > We are currently using 0.18.7
> > 
> > On Feb 27, 6:22 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > 
> > > The setting only applies in 0.19, but you did not answer the question  
> > > with a bit more detailed information (how many indices, how many shards,  
> > > are they not evenly balanced)?
> > 
> > > On Wednesday, February 22, 2012 at 1:32 AM,ChrisRodewrote:
> > > 
> > > > UPDATE: Unfortunately applying this transiently baulked.. however I  
> > > > have applied manually to each node in the cluster. After the dust  
> > > > settled, the shards still aren't distributed fairly.  
> > > > We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
> > > > = 12 so this should be fine) and some nodes still have 20 shards on  
> > > > an  
> > > > index.
> > 
> > > > We are using 18.6 is this an 18.7 only setting by any chance?
> > 
> > > > On Feb 21, 5:16 pm,ChrisRode\<[cir...@gmail.com](mailto:cir...@gmail.com) ([http://gmail.com](http://gmail.com))\>  
> > > > wrote:
> > > > 
> > > > > Hey Shay
> > 
> > > > > Yes we do, we have 3, each with 20 shards. Looking with  
> > > > > elasticsearch-  
> > > > > head I can see that each index is not spread evenly. Trying to use  
> > > > > index.routing.allocation.total\_shards\_per\_node (set to 15)  
> > > > > transiently  
> > > > > now...
> > 
> > > > > Anything else I should try?
> > 
> > > > > On Feb 21, 12:12 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) (  
> > > > > [http://gmail.com](http://gmail.com))\> wrote:
> > 
> > > > > > Do you have multiple indices in the cluster? If so, maybe you  
> > > > > > end up with certain indices that only on a subset of the cluster?
> > 
> > > > > > On Monday, February 20, 2012 at 6:01 AM,ChrisRodewrote:
> > > > > > 
> > > > > > > Hi all
> > 
> > > > > > > I have not been able to find anything to explain why this may  
> > > > > > > be occurring..
> > 
> > > > > > > We have a cluster of 7 nodes on ephemeral EC2 with an S3  
> > > > > > > gateway set to 10 minutes. When we put the cluster under load though, only  
> > > > > > > 3 of the 7 nodes get any load on them, and that quickly builds up to a  
> > > > > > > point where queries are timing out.
> > 
> > > > > > > We originally had a replication factor of 3 set, and have just  
> > > > > > > updated it to 6 (shard count is 20 per index), however this does not seem  
> > > > > > > to have done the trick
> > 
> > > > > > > Any ideas?

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 27, 2012, 1:09pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/11 "2012-02-27T13:09:39Z")

</div>

As Berkay mentioned, the balancing theme is across indices, though I have started to work on smarter one that will do better cross indices case. In 0.19, I added a bandaid (for now) where you can force, on the index level, max number of shards per node, so you can force even balancing (take total\_number\_of\_shards / number\_of\_nodes, where total\_number\_of\_shards is (1+number\_of\_replicas) \* number\_of\_shards).

On Monday, February 27, 2012 at 5:12 AM, Berkay Mollamustafaoglu wrote:

> Chris,
> 
> ES does not distribute shards to accomplish " even number of shards per index". It distributes to have even number of shards per node in total. Currently in ES, when you have indices with drastically different sizes, you end up with unbalanced load.  
> Why do you have a separate index with 20 shards for the index with 106 documents, is it supposed to grow? The easiest way to resolve the problem is to have only a single index with different types for 2nd and 3rd index, rather than having separate indices.
> 
> Regards,  
> Berkay Mollamustafaoglu  
> mberkay on yahoo, google and skype
> 
> On Sun, Feb 26, 2012 at 7:36 PM, Chris Rode \<[cirode@gmail.com](mailto:cirode@gmail.com) ([mailto:cirode@gmail.com](mailto:cirode@gmail.com))\> wrote:
> 
> > This new skitch illustrates what I was saying about the index-node  
> > balancing behavior
> > 
> > [Skitch | Evernote](https://skitch.com/christopherrode/8r9xr/elasticsearch-head)
> > 
> > On Feb 27, 10:25 am, Chris Rode \<[cir...@gmail.com](mailto:cir...@gmail.com) ([mailto:cir...@gmail.com](mailto:cir...@gmail.com))\> wrote:
> > 
> > > Sorry Shay
> > > 
> > > The image attached via skitch shows the cluster and distribution at  
> > > the time of the Jstack trace
> > > 
> > > We have 4 nodes, with 3 indexes and a replication factor of 3. Each  
> > > index has 20 shards. These are running across two availability zones  
> > > on amazon.
> > > 
> > > When we turn down the replication factor the nodes get an even number  
> > > of shards, however they do not get an even number of shards per index.  
> > > This means that indexes with higher load do not utilise the entire  
> > > clusters resources
> > > 
> > > We are currently using 0.18.7
> > > 
> > > On Feb 27, 6:22 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([mailto:kim...@gmail.com](mailto:kim...@gmail.com))\> wrote:
> > > 
> > > > The setting only applies in 0.19, but you did not answer the question with a bit more detailed information (how many indices, how many shards, are they not evenly balanced)?
> > > 
> > > > On Wednesday, February 22, 2012 at 1:32 AM,ChrisRodewrote:
> > > > 
> > > > > UPDATE: Unfortunately applying this transiently baulked.. however I  
> > > > > have applied manually to each node in the cluster. After the dust  
> > > > > settled, the shards still aren't distributed fairly.  
> > > > > We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
> > > > > = 12 so this should be fine) and some nodes still have 20 shards on an  
> > > > > index.
> > > 
> > > > > We are using 18.6 is this an 18.7 only setting by any chance?
> > > 
> > > > > On Feb 21, 5:16 pm,ChrisRode\<[cir...@gmail.com](mailto:cir...@gmail.com) ([mailto:cir...@gmail.com](mailto:cir...@gmail.com)) ([http://gmail.com](http://gmail.com))\> wrote:
> > > > > 
> > > > > > Hey Shay
> > > 
> > > > > > Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
> > > > > > head I can see that each index is not spread evenly. Trying to use  
> > > > > > index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
> > > > > > now...
> > > 
> > > > > > Anything else I should try?
> > > 
> > > > > > On Feb 21, 12:12 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([mailto:kim...@gmail.com](mailto:kim...@gmail.com)) ([http://gmail.com](http://gmail.com))\> wrote:
> > > 
> > > > > > > Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> > > 
> > > > > > > On Monday, February 20, 2012 at 6:01 AM,ChrisRodewrote:
> > > > > > > 
> > > > > > > > Hi all
> > > 
> > > > > > > > I have not been able to find anything to explain why this may be occurring..
> > > 
> > > > > > > > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> > > 
> > > > > > > > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> > > 
> > > > > > > > Any ideas?

---

<div class="post-metadata">

**Author:** ![Nick\_Dimiduk](https://avatars.discourse-cdn.com/v4/letter/n/c77e96/32.png) [@Nick\_Dimiduk](https://discuss.elastic.co/u/Nick_Dimiduk)\
**Post date:** [February 28, 2012, 10:49pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/12 "2012-02-28T22:49:02Z")

</div>

Hi,

I am having a similar experience. In my case, 1 node is pegged while  
others have only 25-50% CPU utilization. That over-burdened node is  
hosting 4 index shards while the others have 3; it is not the master.  
None of the nodes are swapping.

Configuration.

Running v0.18.7. I have only a single index: 5 shards, 1 replica. ES  
Head reports it as ~1mil documents and ~1.8g in size. The index is  
configured to be entirely in memory with the S3 gateway. ES\_MAX\_MEM is  
at 4g. I'm running it on a 3-node cluster of m1.large instances.

Workload.

I started with a bulk-insert of all the data. After that, a couple  
schema tweaks resulted in additional loads of a portion of he data  
set. The querying being run is "match\_all" with a "geo\_distance"  
filter applied. At this point no other fields are being filtered. No  
writes are happening durring this read test.

Why is this single machine the bottle-neck? How can I go about  
distributing this load more evenly? Is there a minimum cluster size or  
specific cluster configuration I should be using for this kind of  
load? I'm curious how I can even go about diagnosing the issue.

Thanks,  
Nick

On Feb 27, 5:09 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> As Berkay mentioned, the balancing theme is across indices, though I have started to work on smarter one that will do better cross indices case. In 0.19, I added a bandaid (for now) where you can force, on the index level, max number of shards per node, so you can force even balancing (take total\_number\_of\_shards / number\_of\_nodes, where total\_number\_of\_shards is (1+number\_of\_replicas) \* number\_of\_shards).
> 
> On Monday, February 27, 2012 at 5:12 AM, Berkay Mollamustafaoglu wrote:
> 
> > Chris,
> 
> > ES does not distribute shards to accomplish " even number of shards per index". It distributes to have even number of shards per node in total. Currently in ES, when you have indices with drastically different sizes, you end up with unbalanced load.  
> > Why do you have a separate index with 20 shards for the index with 106 documents, is it supposed to grow? The easiest way to resolve the problem is to have only a single index with different types for 2nd and 3rd index, rather than having separate indices.
> 
> > Regards,  
> > Berkay Mollamustafaoglu  
> > mberkay on yahoo, google and skype
> 
> > On Sun, Feb 26, 2012 at 7:36 PM, Chris Rode \<[cir...@gmail.com](mailto:cir...@gmail.com) ([mailto:cir...@gmail.com](mailto:cir...@gmail.com))\> wrote:
> > 
> > > This new skitch illustrates what I was saying about the index-node  
> > > balancing behavior
> 
> > > [Skitch | Evernote](https://skitch.com/christopherrode/8r9xr/elasticsearch-head)
> 
> > > On Feb 27, 10:25 am, Chris Rode \<[cir...@gmail.com](mailto:cir...@gmail.com) ([mailto:cir...@gmail.com](mailto:cir...@gmail.com))\> wrote:
> > > 
> > > > Sorry Shay
> 
> > > > The image attached via skitch shows the cluster and distribution at  
> > > > the time of the Jstack trace
> 
> > > > We have 4 nodes, with 3 indexes and a replication factor of 3. Each  
> > > > index has 20 shards. These are running across two availability zones  
> > > > on amazon.
> 
> > > > When we turn down the replication factor the nodes get an even number  
> > > > of shards, however they do not get an even number of shards per index.  
> > > > This means that indexes with higher load do not utilise the entire  
> > > > clusters resources
> 
> > > > We are currently using 0.18.7
> 
> > > > On Feb 27, 6:22 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([mailto:kim...@gmail.com](mailto:kim...@gmail.com))\> wrote:
> 
> > > > > The setting only applies in 0.19, but you did not answer the question with a bit more detailed information (how many indices, how many shards, are they not evenly balanced)?
> 
> > > > > On Wednesday, February 22, 2012 at 1:32 AM,ChrisRodewrote:
> > > > > 
> > > > > > UPDATE: Unfortunately applying this transiently baulked.. however I  
> > > > > > have applied manually to each node in the cluster. After the dust  
> > > > > > settled, the shards still aren't distributed fairly.  
> > > > > > We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
> > > > > > = 12 so this should be fine) and some nodes still have 20 shards on an  
> > > > > > index.
> 
> > > > > > We are using 18.6 is this an 18.7 only setting by any chance?
> 
> > > > > > On Feb 21, 5:16 pm,ChrisRode\<[cir...@gmail.com](mailto:cir...@gmail.com) ([mailto:cir...@gmail.com](mailto:cir...@gmail.com)) ([http://gmail.com](http://gmail.com))\> wrote:
> > > > > > 
> > > > > > > Hey Shay
> 
> > > > > > > Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
> > > > > > > head I can see that each index is not spread evenly. Trying to use  
> > > > > > > index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
> > > > > > > now...
> 
> > > > > > > Anything else I should try?
> 
> > > > > > > On Feb 21, 12:12 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([mailto:kim...@gmail.com](mailto:kim...@gmail.com)) ([http://gmail.com](http://gmail.com))\> wrote:
> 
> > > > > > > > Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> 
> > > > > > > > On Monday, February 20, 2012 at 6:01 AM,ChrisRodewrote:
> > > > > > > > 
> > > > > > > > > Hi all
> 
> > > > > > > > > I have not been able to find anything to explain why this may be occurring..
> 
> > > > > > > > > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> 
> > > > > > > > > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> 
> > > > > > > > > Any ideas?

---

<div class="post-metadata">

**Author:** ![Chris\_Rode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_rode/32/2923_2.png) [@Chris\_Rode](https://discuss.elastic.co/u/Chris_Rode)\
**Post date:** [February 29, 2012, 3:12am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/13 "2012-02-29T03:12:30Z")

</div>

Thanks for that Shay

(I was wondering why that setting wasnt taking effect on 0.18.7 :)).  
That setting will help distribute the shards in the multi-index case.  
For now we have simply combined our indexes. We are now down to 2 and  
it is more evently distributed due to that change

We ended up getting the cluster to cope with the amount of reads by  
switching on the caching for certain filters. So far this is working  
for us... although before the cache warms we still get the problem  
that Nick Mentions below, where one or two machines get high load and  
the rest dont show any.. this disapears as the cache gets populated

On Feb 28, 12:09 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> As Berkay mentioned, the balancing theme is across indices, though I have started to work on smarter one that will do better cross indices case. In 0.19, I added a bandaid (for now) where you can force, on the index level, max number of shards per node, so you can force even balancing (take total\_number\_of\_shards / number\_of\_nodes, where total\_number\_of\_shards is (1+number\_of\_replicas) \* number\_of\_shards).
> 
> On Monday, February 27, 2012 at 5:12 AM, Berkay Mollamustafaoglu wrote:
> 
> > Chris,
> 
> > ES does not distribute shards to accomplish " even number of shards per index". It distributes to have even number of shards per node in total. Currently in ES, when you have indices with drastically different sizes, you end up with unbalanced load.  
> > Why do you have a separate index with 20 shards for the index with 106 documents, is it supposed to grow? The easiest way to resolve the problem is to have only a single index with different types for 2nd and 3rd index, rather than having separate indices.
> 
> > Regards,  
> > Berkay Mollamustafaoglu  
> > mberkay on yahoo, google and skype
> 
> > On Sun, Feb 26, 2012 at 7:36 PM, Chris Rode \<[cir...@gmail.com](mailto:cir...@gmail.com) ([mailto:cir...@gmail.com](mailto:cir...@gmail.com))\> wrote:
> > 
> > > This new skitch illustrates what I was saying about the index-node  
> > > balancing behavior
> 
> > > [Skitch | Evernote](https://skitch.com/christopherrode/8r9xr/elasticsearch-head)
> 
> > > On Feb 27, 10:25 am, Chris Rode \<[cir...@gmail.com](mailto:cir...@gmail.com) ([mailto:cir...@gmail.com](mailto:cir...@gmail.com))\> wrote:
> > > 
> > > > Sorry Shay
> 
> > > > The image attached via skitch shows the cluster and distribution at  
> > > > the time of the Jstack trace
> 
> > > > We have 4 nodes, with 3 indexes and a replication factor of 3. Each  
> > > > index has 20 shards. These are running across two availability zones  
> > > > on amazon.
> 
> > > > When we turn down the replication factor the nodes get an even number  
> > > > of shards, however they do not get an even number of shards per index.  
> > > > This means that indexes with higher load do not utilise the entire  
> > > > clusters resources
> 
> > > > We are currently using 0.18.7
> 
> > > > On Feb 27, 6:22 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([mailto:kim...@gmail.com](mailto:kim...@gmail.com))\> wrote:
> 
> > > > > The setting only applies in 0.19, but you did not answer the question with a bit more detailed information (how many indices, how many shards, are they not evenly balanced)?
> 
> > > > > On Wednesday, February 22, 2012 at 1:32 AM,ChrisRodewrote:
> > > > > 
> > > > > > UPDATE: Unfortunately applying this transiently baulked.. however I  
> > > > > > have applied manually to each node in the cluster. After the dust  
> > > > > > settled, the shards still aren't distributed fairly.  
> > > > > > We have set the setting to 13 (20(shards)\*4(original+copies)/7(nodes)  
> > > > > > = 12 so this should be fine) and some nodes still have 20 shards on an  
> > > > > > index.
> 
> > > > > > We are using 18.6 is this an 18.7 only setting by any chance?
> 
> > > > > > On Feb 21, 5:16 pm,ChrisRode\<[cir...@gmail.com](mailto:cir...@gmail.com) ([mailto:cir...@gmail.com](mailto:cir...@gmail.com)) ([http://gmail.com](http://gmail.com))\> wrote:
> > > > > > 
> > > > > > > Hey Shay
> 
> > > > > > > Yes we do, we have 3, each with 20 shards. Looking with elasticsearch-  
> > > > > > > head I can see that each index is not spread evenly. Trying to use  
> > > > > > > index.routing.allocation.total\_shards\_per\_node (set to 15) transiently  
> > > > > > > now...
> 
> > > > > > > Anything else I should try?
> 
> > > > > > > On Feb 21, 12:12 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([mailto:kim...@gmail.com](mailto:kim...@gmail.com)) ([http://gmail.com](http://gmail.com))\> wrote:
> 
> > > > > > > > Do you have multiple indices in the cluster? If so, maybe you end up with certain indices that only on a subset of the cluster?
> 
> > > > > > > > On Monday, February 20, 2012 at 6:01 AM,ChrisRodewrote:
> > > > > > > > 
> > > > > > > > > Hi all
> 
> > > > > > > > > I have not been able to find anything to explain why this may be occurring..
> 
> > > > > > > > > We have a cluster of 7 nodes on ephemeral EC2 with an S3 gateway set to 10 minutes. When we put the cluster under load though, only 3 of the 7 nodes get any load on them, and that quickly builds up to a point where queries are timing out.
> 
> > > > > > > > > We originally had a replication factor of 3 set, and have just updated it to 6 (shard count is 20 per index), however this does not seem to have done the trick
> 
> > > > > > > > > Any ideas?

---

<div class="post-metadata">

**Author:** ![Frederic](https://avatars.discourse-cdn.com/v4/letter/f/4491bb/32.png) [@Frederic](https://discuss.elastic.co/u/Frederic)\
**Post date:** [February 29, 2012, 12:55pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/14 "2012-02-29T12:55:06Z")

</div>

Hi there,

For what it worths, I have the same issue here. We upgraded our cluster to  
0.18.7 (6 servers, 1 index, 20 shards, 1 replica) two days ago and set max  
and min heap as 4GB. Afterwards we started to index 50G of docs at about  
150 docs/sec, and one of the servers showed really high loads (between 20  
and 40) and CPU usage was 200% in a 4 cores CPU. After restarting that  
server, other server showed similar behavior.

Thanks,  
Frederic

On Monday, 27 February 2012 10:09:39 UTC-3, kimchy wrote:

> As Berkay mentioned, the balancing theme is across indices, though I have  
> started to work on smarter one that will do better cross indices case. In  
> 0.19, I added a bandaid (for now) where you can force, on the index level,  
> max number of shards per node, so you can force even balancing (take  
> total\_number\_of\_shards / number\_of\_nodes, where total\_number\_of\_shards is  
> (1+number\_of\_replicas) \* number\_of\_shards).
> 
> On Monday, February 27, 2012 at 5:12 AM, Berkay Mollamustafaoglu wrote:

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:37am UTC](https://discuss.elastic.co/t/high-cpu-load-on-only-some-of-the-machines-in-a-cluster/6751/15 "2017-07-06T03:37:52Z")

</div>