Decommission of multiple nodes

Hi,

I wanted to reduce my cluster size by 3 data nodes (from 27 to 24).
I issued 3 decommission API calls one after the other and saw their shards
start moving.
I waited for all the shards to relocate, but when I checked to make sure
they can safely be removed I found out that they have new shards on them.

Is there a limit to the number of nodes that can be decommissioned at the
same time?
Is there a timeout for the decommission definition for a node?
Anything else?

Thanks for the help,
Noam

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/86afdf02-3dff-4495-a555-8f70f9356216%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

What do you mean by decommission, there is no API call for that, unless you
meant shutdown?

The easiest way would be to power down one node at a time, waiting for
things to rebalance before shutting another down.

On 3 December 2014 at 03:01, Noam Gazit noam@totango.com wrote:

Hi,

I wanted to reduce my cluster size by 3 data nodes (from 27 to 24).
I issued 3 decommission API calls one after the other and saw their shards
start moving.
I waited for all the shards to relocate, but when I checked to make sure
they can safely be removed I found out that they have new shards on them.

Is there a limit to the number of nodes that can be decommissioned at the
same time?
Is there a timeout for the decommission definition for a node?
Anything else?

Thanks for the help,
Noam

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/86afdf02-3dff-4495-a555-8f70f9356216%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/86afdf02-3dff-4495-a555-8f70f9356216%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8%3DLa7uK-Aiykx5Xkm0hy3-dLuY7L_0QOzLRzApCxHQGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

If you mean allocation filtering (like
cluster.routing.allocation.exclude._ip) then you just need to specify all
three ip addresses with commas between them.

Nik

On Tue, Dec 2, 2014 at 4:00 PM, Mark Walkom markwalkom@gmail.com wrote:

What do you mean by decommission, there is no API call for that, unless
you meant shutdown?

The easiest way would be to power down one node at a time, waiting for
things to rebalance before shutting another down.

On 3 December 2014 at 03:01, Noam Gazit noam@totango.com wrote:

Hi,

I wanted to reduce my cluster size by 3 data nodes (from 27 to 24).
I issued 3 decommission API calls one after the other and saw their
shards start moving.
I waited for all the shards to relocate, but when I checked to make sure
they can safely be removed I found out that they have new shards on them.

Is there a limit to the number of nodes that can be decommissioned at the
same time?
Is there a timeout for the decommission definition for a node?
Anything else?

Thanks for the help,
Noam

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/86afdf02-3dff-4495-a555-8f70f9356216%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/86afdf02-3dff-4495-a555-8f70f9356216%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8%3DLa7uK-Aiykx5Xkm0hy3-dLuY7L_0QOzLRzApCxHQGg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8%3DLa7uK-Aiykx5Xkm0hy3-dLuY7L_0QOzLRzApCxHQGg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1i8H%3D6A4utfWh7EVSDg_OK6RdvHDKVgkWXfFEeOt%2BG1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi Nik,

I found the that I am not able to do decommissioning over multiple nodes.
I can do decommissioning over a single node.

These cmds below work perfectly for my ES cluster:
"cluster.routing.allocation.exclude._name": "node-1"
"cluster.routing.allocation.exclude._name": "node-2"
"cluster.routing.allocation.exclude._ip": "ip1"
"cluster.routing.allocation.exclude._ip": "ip2"

But none of these cmds below has effect for my ES cluster:
"cluster.routing.allocation.exclude._name": "node-1,node-2"
"cluster.routing.allocation.exclude._name": ["node-1", "node-2"]
"cluster.routing.allocation.exclude._ip": "ip1,ip2"
"cluster.routing.allocation.exclude._ip": "ip1, ip2"

I also notice that extra spaces for "_name" or "_ip" field cause the cmd to
fail (no effect).
For example, none of these cmds below works.
"cluster.routing.allocation.exclude._ip": " ip1 "
"cluster.routing.allocation.exclude._name": "node-1 "

I am running this version of ES on Ubuntu 14.04:

elasticsearch -v
Version: 1.4.0, Build: bc94bd8/2014-11-05T14:26:12Z, JVM: 1.8.0_25

I checked release notes of ES 1.4.1 and 1.4.2, and I didn't find some bug
fixing notes for this "decommissioning over multiple nodes" function.

Is this a bug? Or, I omit something important?

Regards,
Ye

On Tuesday, December 2, 2014 4:07:15 PM UTC-5, Nikolas Everett wrote:

If you mean allocation filtering (like
cluster.routing.allocation.exclude._ip) then you just need to specify all
three ip addresses with commas between them.

Nik

On Tue, Dec 2, 2014 at 4:00 PM, Mark Walkom <markw...@gmail.com
<javascript:>> wrote:

What do you mean by decommission, there is no API call for that, unless
you meant shutdown?

The easiest way would be to power down one node at a time, waiting for
things to rebalance before shutting another down.

On 3 December 2014 at 03:01, Noam Gazit <no...@totango.com <javascript:>>
wrote:

Hi,

I wanted to reduce my cluster size by 3 data nodes (from 27 to 24).
I issued 3 decommission API calls one after the other and saw their
shards start moving.
I waited for all the shards to relocate, but when I checked to make sure
they can safely be removed I found out that they have new shards on them.

Is there a limit to the number of nodes that can be decommissioned at
the same time?
Is there a timeout for the decommission definition for a node?
Anything else?

Thanks for the help,
Noam

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/86afdf02-3dff-4495-a555-8f70f9356216%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/86afdf02-3dff-4495-a555-8f70f9356216%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8%3DLa7uK-Aiykx5Xkm0hy3-dLuY7L_0QOzLRzApCxHQGg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8%3DLa7uK-Aiykx5Xkm0hy3-dLuY7L_0QOzLRzApCxHQGg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/820fb54b-d932-4a7f-8ee5-906b88a5e2b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Wed, Dec 17, 2014 at 6:03 PM, Ye D ye@volarvideo.com wrote:

"cluster.routing.allocation.exclude._ip": "ip1, ip2"

I use this one and I'm pretty sure its worked for me in the past.

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3_DpdkrMgGMiNGJEaorhc55N3znc9Wx%2BA2HC%2BocHFJ0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

1 Like