Quick recovery after node restart in elasticsearch

Consider the below settings in the elasticsearch.yml

gateway.recover_after_data_nodes: 3
gateway.recover_after_time: 5m
gateway.expected_data_nodes: 3

Current setting: Say, I have 3 data nodes. Now if I decide to restart a data node(due to a small change in setting), the recovery will start immediately after node restart as per the expected_data_nodes setting. There will be many unassigned shards, which will get allocated slowly depending on the data it contains.

In order to avoid that, is there any way to allocate all the unassigned shards to a specific node?(in my case the restarted node) and once that is done, ES should take over the rebalancing.

Mainly I want to avoid the heavy timelag of the cluster state from yellow to green.(it is in the range of hours in my case)

Can I use the cluster reroute api for this purpose?

or is there any other api to transfer all the unassigned shards to specific node at one go?

My procedure to restart a node without shards reallocation is :

curl -XPUT localhost:9200/_cluster/settings -d
'{"transient":{"cluster.routing.allocation.disable_allocation": true}}'
/etc/init.d/elasticsearch restart
curl -XPUT localhost:9200/_cluster/settings -d
'{"transient":{"cluster.routing.allocation.disable_allocation": false}}'

Shard keep analocated until
"cluster.routing.allocation.disable_allocation": false, then shards recover
on the server just restarted (starting at size they were before shutdown)
It is very quick.

Cheers,

--
Damien

2013/4/26 ovamsikrishna ovamsikrishna@gmail.com

Consider the below settings in the elasticsearch.yml

gateway.recover_after_data_nodes: 3
gateway.recover_after_time: 5m
gateway.expected_data_nodes: 3

Current setting: Say, I have 3 data nodes. Now if I decide to restart a
data
node(due to a small change in setting), the recovery will start immediately
after node restart as per the expected_data_nodes setting. There will be
many unassigned shards, which will get allocated slowly depending on the
data it contains.

In order to avoid that, is there any way to allocate all the unassigned
shards to a specific node?(in my case the restarted node) and once that is
done, ES should take over the rebalancing.

Mainly I want to avoid the heavy timelag of the cluster state from yellow
to
green.(it is in the range of hours in my case)

Can I use the cluster reroute api for this purpose?

or is there any other api to transfer all the unassigned shards to specific
node at one go?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/quick-recovery-after-node-restart-in-elasticsearch-tp4033876.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Another option is to prepare the node ahead of time by excluding it using
the shard placement API. The cluster will immediately begin moving shards
off the node until it has none left, allowing you to restart it as many
times as you want until it is ready for use again. Depending on why you
need to restart it, that may be more useful to you as it prevents your
cluster from every going into a degraded state at all.

-Pradeep

On Friday, April 26, 2013 1:25:31 AM UTC-7, Damien Hardy wrote:

My procedure to restart a node without shards reallocation is :

curl -XPUT localhost:9200/_cluster/settings -d
'{"transient":{"cluster.routing.allocation.disable_allocation": true}}'
/etc/init.d/elasticsearch restart
curl -XPUT localhost:9200/_cluster/settings -d
'{"transient":{"cluster.routing.allocation.disable_allocation": false}}'

Shard keep analocated until
"cluster.routing.allocation.disable_allocation": false, then shards recover
on the server just restarted (starting at size they were before shutdown)
It is very quick.

Cheers,

--
Damien

2013/4/26 ovamsikrishna <ovamsi...@gmail.com <javascript:>>

Consider the below settings in the elasticsearch.yml

gateway.recover_after_data_nodes: 3
gateway.recover_after_time: 5m
gateway.expected_data_nodes: 3

Current setting: Say, I have 3 data nodes. Now if I decide to restart a
data
node(due to a small change in setting), the recovery will start
immediately
after node restart as per the expected_data_nodes setting. There will be
many unassigned shards, which will get allocated slowly depending on the
data it contains.

In order to avoid that, is there any way to allocate all the unassigned
shards to a specific node?(in my case the restarted node) and once that is
done, ES should take over the rebalancing.

Mainly I want to avoid the heavy timelag of the cluster state from yellow
to
green.(it is in the range of hours in my case)

Can I use the cluster reroute api for this purpose?

or is there any other api to transfer all the unassigned shards to
specific
node at one go?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/quick-recovery-after-node-restart-in-elasticsearch-tp4033876.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks a lot for all the replies. Will try the above allocation curl command next time before node restart...

Before that I need a small clarification. This command seems to be affecting entire cluster. But, in my case I want that to be done for a single node. Is that possible?

From the command it seems, the there will not be any further allocation in the entire cluster till we set it to false. I don't want like that. The allocation should be stopped only for that node. After restart everything should go into normal state in a faster pace.

Is it possible to specify the node in the above command?

On Fri, Apr 26, 2013 at 10:25:31AM +0200, Damien Hardy wrote:

Damien,

My procedure to restart a node without shards reallocation is :

curl -XPUT localhost:9200/_cluster/settings -d
'{"transient":{"cluster.routing.allocation.disable_allocation": true}}'
/etc/init.d/elasticsearch restart
curl -XPUT localhost:9200/_cluster/settings -d
'{"transient":{"cluster.routing.allocation.disable_allocation": false}}'

Shard keep analocated until
"cluster.routing.allocation.disable_allocation": false, then shards recover
on the server just restarted (starting at size they were before shutdown)
It is very quick.

I am quite new to elasticsearch, so this may be kind of newbeish. Does this
also mean, if a primary shard would reside on that node, replica promotion will
not happen?

Best regards, Adrian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.