Wrong load distribution

Hi all,

I meet a load distribution problem today and I browsed the Internet to find
out someone having the same problem but unfortunately, no one seems to
have. Here is my cluster configuration:

  • 4 machines with 16 Go Ram (9600 Mo allocated to JVM)
  • 8 core CPUs
  • Special cluster config: awareness of the vm host for shard primary and
    replica allocation (to avoid having both a primary and a replica on the
    same hardware).

I tried a (huge) insert today, and it leads to one of the 4 nodes having a
load average extremely higher than the three others (1.5 to 2 for the 3
"relax" servers vs. >12 for the high loaded one). I was thinking that
Elasticsearch was designed to avoid it. Am I wrong here?

Any tips is welcome :slight_smile:

Regards,
Loïc

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Do you have your shards equally distributed over the 4 nodes? Or do you use
the default of 5 shards?

Jörg

On Tue, Mar 31, 2015 at 5:28 PM, Loïc Wenkin loic.wenkin@gmail.com wrote:

Hi all,

I meet a load distribution problem today and I browsed the Internet to
find out someone having the same problem but unfortunately, no one seems to
have. Here is my cluster configuration:

  • 4 machines with 16 Go Ram (9600 Mo allocated to JVM)
  • 8 core CPUs
  • Special cluster config: awareness of the vm host for shard primary and
    replica allocation (to avoid having both a primary and a replica on the
    same hardware).

I tried a (huge) insert today, and it leads to one of the 4 nodes having a
load average extremely higher than the three others (1.5 to 2 for the 3
"relax" servers vs. >12 for the high loaded one). I was thinking that
Elasticsearch was designed to avoid it. Am I wrong here?

Any tips is welcome :slight_smile:

Regards,
Loïc

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-iy-%3D5NPb0YoYuydwd-MXR%2BuLQ7p_4m9mNVrNLyEPyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi Jörg,

Thanks for your reply. I have the default 5 shards, but I have multiple
indexes, and all nodes have about the same number of shards on them.

Regards,
Loïc

Le mercredi 1 avril 2015 00:03:06 UTC+2, Jörg Prante a écrit :

Do you have your shards equally distributed over the 4 nodes? Or do you
use the default of 5 shards?

Jörg

On Tue, Mar 31, 2015 at 5:28 PM, Loïc Wenkin <loic....@gmail.com
<javascript:>> wrote:

Hi all,

I meet a load distribution problem today and I browsed the Internet to
find out someone having the same problem but unfortunately, no one seems to
have. Here is my cluster configuration:

  • 4 machines with 16 Go Ram (9600 Mo allocated to JVM)
  • 8 core CPUs
  • Special cluster config: awareness of the vm host for shard primary and
    replica allocation (to avoid having both a primary and a replica on the
    same hardware).

I tried a (huge) insert today, and it leads to one of the 4 nodes having
a load average extremely higher than the three others (1.5 to 2 for the 3
"relax" servers vs. >12 for the high loaded one). I was thinking that
Elasticsearch was designed to avoid it. Am I wrong here?

Any tips is welcome :slight_smile:

Regards,
Loïc

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4abd9e01-b4b3-4b5c-9ccf-66853c84b255%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You should set k*n shards for your indexes to avoid balancing troubles
with k being a constant integer and n being the number of nodes.

Also keep an eye on dynamic mapping. if you have add thousands of new
field names steadily over time, you exercise the master node.

Jörg

On Wed, Apr 1, 2015 at 9:21 AM, Loïc Wenkin loic.wenkin@gmail.com wrote:

Hi Jörg,

Thanks for your reply. I have the default 5 shards, but I have multiple
indexes, and all nodes have about the same number of shards on them.

Regards,
Loïc

Le mercredi 1 avril 2015 00:03:06 UTC+2, Jörg Prante a écrit :

Do you have your shards equally distributed over the 4 nodes? Or do you
use the default of 5 shards?

Jörg

On Tue, Mar 31, 2015 at 5:28 PM, Loïc Wenkin loic....@gmail.com wrote:

Hi all,

I meet a load distribution problem today and I browsed the Internet to
find out someone having the same problem but unfortunately, no one seems to
have. Here is my cluster configuration:

  • 4 machines with 16 Go Ram (9600 Mo allocated to JVM)
  • 8 core CPUs
  • Special cluster config: awareness of the vm host for shard primary and
    replica allocation (to avoid having both a primary and a replica on the
    same hardware).

I tried a (huge) insert today, and it leads to one of the 4 nodes having
a load average extremely higher than the three others (1.5 to 2 for the 3
"relax" servers vs. >12 for the high loaded one). I was thinking that
Elasticsearch was designed to avoid it. Am I wrong here?

Any tips is welcome :slight_smile:

Regards,
Loïc

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4abd9e01-b4b3-4b5c-9ccf-66853c84b255%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4abd9e01-b4b3-4b5c-9ccf-66853c84b255%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGdqp5jnzJXis6J82fN%2BwRaJAcAWUpQr7OFi24cVQwgLQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.