Different hardware capacity

Hello,
On one of my ELS cluster, i have node with different hardware capacity.
1 node : 8 GB RAM and 200GB disk
1 node : 4 GB RAM and 20GB disk
2 node : 64GB RAM with 4To Disk

I find that ELS tries to balance the same amount of data on each node.
The 2 smaller node are near full (disks and cpu) while the 2 biggers don't
do much work.
And so they crash often with OOM or others errors.

Is there any parameters like in hadoop to have the data distributed by %
instead of MB and so with the memory ?

regards

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4014e188-458c-429b-b6c2-7af941a8302e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You can balance, to a degree, based on disk space, but not heap/system RAM.
There might be other options, like playing with shard allocation.

See
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html
for some ideas.

On 11 November 2014 19:43, lagarutte via elasticsearch <
elasticsearch@googlegroups.com> wrote:

Hello,
On one of my ELS cluster, i have node with different hardware capacity.
1 node : 8 GB RAM and 200GB disk
1 node : 4 GB RAM and 20GB disk
2 node : 64GB RAM with 4To Disk

I find that ELS tries to balance the same amount of data on each node.
The 2 smaller node are near full (disks and cpu) while the 2 biggers don't
do much work.
And so they crash often with OOM or others errors.

Is there any parameters like in hadoop to have the data distributed by %
instead of MB and so with the memory ?

regards

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4014e188-458c-429b-b6c2-7af941a8302e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4014e188-458c-429b-b6c2-7af941a8302e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DL6ag3GZtz9Zh30%3Do9CHKyPsyQ%2B5Vi%2BtNEw7iH9DKe-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch doesn't let you weight nodes for balance and the disk space
allocation decider really just puts soft limits on the amount of space
elasticsearch can take up per machine. There really isn't anything to do
it automatically.

You could use a combination of allocation awareness, total_shards_per_node,
and forcibly pinning the shards to machines to get something. Its the best
there is right now.

On Tue, Nov 11, 2014 at 3:46 AM, Mark Walkom markwalkom@gmail.com wrote:

You can balance, to a degree, based on disk space, but not heap/system RAM.
There might be other options, like playing with shard allocation.

See
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html
for some ideas.

On 11 November 2014 19:43, lagarutte via elasticsearch <
elasticsearch@googlegroups.com> wrote:

Hello,
On one of my ELS cluster, i have node with different hardware capacity.
1 node : 8 GB RAM and 200GB disk
1 node : 4 GB RAM and 20GB disk
2 node : 64GB RAM with 4To Disk

I find that ELS tries to balance the same amount of data on each node.
The 2 smaller node are near full (disks and cpu) while the 2 biggers
don't do much work.
And so they crash often with OOM or others errors.

Is there any parameters like in hadoop to have the data distributed by %
instead of MB and so with the memory ?

regards

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4014e188-458c-429b-b6c2-7af941a8302e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4014e188-458c-429b-b6c2-7af941a8302e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DL6ag3GZtz9Zh30%3Do9CHKyPsyQ%2B5Vi%2BtNEw7iH9DKe-Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DL6ag3GZtz9Zh30%3Do9CHKyPsyQ%2B5Vi%2BtNEw7iH9DKe-Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1rpQTzGs0VRbxTTTeF97RKYbcRWa%3DMY3-PuS0veaE8wQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

ok, thanks,so if i understand, the best is to have the same hardware
capacity for all the nodes involved in the cluster.
ELS need more polish with this, perhaps it will come later

Le mardi 11 novembre 2014 14:30:44 UTC+1, Nikolas Everett a écrit :

Elasticsearch doesn't let you weight nodes for balance and the disk space
allocation decider really just puts soft limits on the amount of space
elasticsearch can take up per machine. There really isn't anything to do
it automatically.

You could use a combination of allocation awareness,
total_shards_per_node, and forcibly pinning the shards to machines to get
something. Its the best there is right now.

On Tue, Nov 11, 2014 at 3:46 AM, Mark Walkom <markw...@gmail.com
<javascript:>> wrote:

You can balance, to a degree, based on disk space, but not heap/system
RAM.
There might be other options, like playing with shard allocation.

See
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html
for some ideas.

On 11 November 2014 19:43, lagarutte via elasticsearch <
elasti...@googlegroups.com <javascript:>> wrote:

Hello,
On one of my ELS cluster, i have node with different hardware capacity.
1 node : 8 GB RAM and 200GB disk
1 node : 4 GB RAM and 20GB disk
2 node : 64GB RAM with 4To Disk

I find that ELS tries to balance the same amount of data on each node.
The 2 smaller node are near full (disks and cpu) while the 2 biggers
don't do much work.
And so they crash often with OOM or others errors.

Is there any parameters like in hadoop to have the data distributed by %
instead of MB and so with the memory ?

regards

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4014e188-458c-429b-b6c2-7af941a8302e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4014e188-458c-429b-b6c2-7af941a8302e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DL6ag3GZtz9Zh30%3Do9CHKyPsyQ%2B5Vi%2BtNEw7iH9DKe-Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DL6ag3GZtz9Zh30%3Do9CHKyPsyQ%2B5Vi%2BtNEw7iH9DKe-Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5166eabb-8763-41dd-8ac6-399f9522cfd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.