# Reuse of replicas on cluster restart **URL:** \ **Category:** Elasticsearch\ **Created:** [September 27, 2013, 12:43am UTC](https://discuss.elastic.co/t/reuse-of-replicas-on-cluster-restart/13781 "2013-09-27T00:43:10Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Andrew\_Voelker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_voelker/32/2082_2.png) [@Andrew\_Voelker](https://discuss.elastic.co/u/Andrew_Voelker)\ **Post date:** [September 27, 2013, 12:43am UTC](https://discuss.elastic.co/t/reuse-of-replicas-on-cluster-restart/13781/1 "2013-09-27T00:43:10Z")
When doing maintenance on a cluster [example: minor version ES upgrade on all nodes], the cluster is first shutdown, the maintenance is performed, and ES on the nodes are started in quick succession. The cluster returns to yellow in a few minutes, but it takes hours to get back to green. The reason is it's re-replicating all the primary shards. For this example, there's one replica per shard on all the indexes. Is there an ES setting or patch that would have ES re-use the replicas on the nodes at cluster restart? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 27, 2013, 12:49am UTC](https://discuss.elastic.co/t/reuse-of-replicas-on-cluster-restart/13781/2 "2013-09-27T00:49:09Z")
curl -XPUT node-name:9200/\_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable\_allocation": true}}'; That'll do what you want, run it before restarting the nodes and then again with false when all nodes are restarted. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 27 September 2013 10:43, Andrew Voelker [avoelker@gmail.com](mailto:avoelker@gmail.com) wrote: > When doing maintenance on a cluster [example: minor version ES upgrade on > all nodes], the cluster is first shutdown, the maintenance is performed, > and ES on the nodes are started in quick succession. The cluster returns to > yellow in a few minutes, but it takes hours to get back to green. The > reason is it's re-replicating all the primary shards. For this example, > there's one replica per shard on all the indexes. Is there an ES setting > or patch that would have ES re-use the replicas on the nodes at cluster > restart? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andrew\_Voelker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_voelker/32/2082_2.png) [@Andrew\_Voelker](https://discuss.elastic.co/u/Andrew_Voelker)\ **Post date:** [September 27, 2013, 1:11am UTC](https://discuss.elastic.co/t/reuse-of-replicas-on-cluster-restart/13781/3 "2013-09-27T01:11:49Z")
Hey Mark, thanks for your quick response. I'm a bit confused, though -- on a cluster restart, the transient settings will be forgotten, and I would think the nodes would come up with all shards (primary and replica) still "initializing". Are you suggesting that by setting the disable\_allocation = true before the cluster shutdown, the replica shards on disk will be used on the following restart? On Thursday, September 26, 2013 5:49:09 PM UTC-7, Mark Walkom wrote: > curl -XPUT node-name:9200/\_cluster/settings -d > '{"transient":{"cluster.routing.allocation.disable\_allocation": true}}'; > > That'll do what you want, run it before restarting the nodes and then > again with false when all nodes are restarted. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 27 September 2013 10:43, Andrew Voelker \<[avoe...@gmail.com](mailto:avoe...@gmail.com)\ > > > wrote: > > > When doing maintenance on a cluster [example: minor version ES upgrade on > > all nodes], the cluster is first shutdown, the maintenance is performed, > > and ES on the nodes are started in quick succession. The cluster returns to > > yellow in a few minutes, but it takes hours to get back to green. The > > reason is it's re-replicating all the primary shards. For this example, > > there's one replica per shard on all the indexes. Is there an ES setting > > or patch that would have ES re-use the replicas on the nodes at cluster > > restart? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 27, 2013, 10:03am UTC](https://discuss.elastic.co/t/reuse-of-replicas-on-cluster-restart/13781/4 "2013-09-27T10:03:47Z")
We generally do a rolling restart of the cluster so the transitory nature doesn't impact us. But if you do keep at least one master active, any nodes that rejoin the cluster will initialise the shards locally and minimise any reallocation. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 27 September 2013 11:11, Andrew Voelker [avoelker@gmail.com](mailto:avoelker@gmail.com) wrote: > Hey Mark, thanks for your quick response. I'm a bit confused, though -- > on a cluster restart, the transient settings will be forgotten, and I would > think the nodes would come up with all shards (primary and replica) still > "initializing". Are you suggesting that by setting the disable\_allocation = > true before the cluster shutdown, the replica shards on disk will be used > on the following restart? > > On Thursday, September 26, 2013 5:49:09 PM UTC-7, Mark Walkom wrote: > > > curl -XPUT node-name:9200/_cluster/\*\*settings -d '{"transient":{"cluster. > > \*\*routing.allocation.disable_\*\*allocation": true}}'; > > > > That'll do what you want, run it before restarting the nodes and then > > again with false when all nodes are restarted. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 27 September 2013 10:43, Andrew Voelker [avoe...@gmail.com](mailto:avoe...@gmail.com) wrote: > > > > > When doing maintenance on a cluster [example: minor version ES upgrade > > > on all nodes], the cluster is first shutdown, the maintenance is performed, > > > and ES on the nodes are started in quick succession. The cluster returns to > > > yellow in a few minutes, but it takes hours to get back to green. The > > > reason is it's re-replicating all the primary shards. For this example, > > > there's one replica per shard on all the indexes. Is there an ES > > > setting or patch that would have ES re-use the replicas on the nodes at > > > cluster restart? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andrew\_Voelker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_voelker/32/2082_2.png) [@Andrew\_Voelker](https://discuss.elastic.co/u/Andrew_Voelker)\ **Post date:** [September 27, 2013, 7:17pm UTC](https://discuss.elastic.co/t/reuse-of-replicas-on-cluster-restart/13781/5 "2013-09-27T19:17:59Z")
Unfortunately, disabling allocations around a node restart did not help. The replicas were still re-replicated, after the node was restarted, rejoined the cluster, and allocations were re-enabled. Does anyone else have an idea on how to force a node to re-use replica shards that were on a node prior to the node's shutdown and restart? Regards, Andrew On Friday, September 27, 2013 3:03:47 AM UTC-7, Mark Walkom wrote: > We generally do a rolling restart of the cluster so the transitory nature > doesn't impact us. > > But if you do keep at least one master active, any nodes that rejoin the > cluster will initialise the shards locally and minimise any reallocation. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 27 September 2013 11:11, Andrew Voelker \<[avoe...@gmail.com](mailto:avoe...@gmail.com)\ > > > wrote: > > > Hey Mark, thanks for your quick response. I'm a bit confused, though -- > > on a cluster restart, the transient settings will be forgotten, and I would > > think the nodes would come up with all shards (primary and replica) still > > "initializing". Are you suggesting that by setting the disable\_allocation = > > true before the cluster shutdown, the replica shards on disk will be used > > on the following restart? > > > > On Thursday, September 26, 2013 5:49:09 PM UTC-7, Mark Walkom wrote: > > > > > curl -XPUT node-name:9200/_cluster/\*\*settings -d > > > '{"transient":{"cluster.\*\*routing.allocation.disable_\*\*allocation": > > > true}}'; > > > > > > That'll do what you want, run it before restarting the nodes and then > > > again with false when all nodes are restarted. > > > > > > Regards, > > > Mark Walkom > > > > > > Infrastructure Engineer > > > Campaign Monitor > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > On 27 September 2013 10:43, Andrew Voelker [avoe...@gmail.com](mailto:avoe...@gmail.com) wrote: > > > > > > > When doing maintenance on a cluster [example: minor version ES upgrade > > > > on all nodes], the cluster is first shutdown, the maintenance is performed, > > > > and ES on the nodes are started in quick succession. The cluster returns to > > > > yellow in a few minutes, but it takes hours to get back to green. The > > > > reason is it's re-replicating all the primary shards. For this example, > > > > there's one replica per shard on all the indexes. Is there an ES > > > > setting or patch that would have ES re-use the replicas on the nodes at > > > > cluster restart? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![roytmana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roytmana/32/44855_2.png) [@roytmana](https://discuss.elastic.co/u/roytmana)\ **Post date:** [September 27, 2013, 9:16pm UTC](https://discuss.elastic.co/t/reuse-of-replicas-on-cluster-restart/13781/6 "2013-09-27T21:16:30Z")
It would be nice if it was possible to put custer in sercuce mode so it qould reject all requests and then shut it down safely Then upgrade and start all the nodes so they would start in service mode and then issue command to exit service mode which would then recover all shards with hopefully nothing to recover as everything would be in synch -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:14am UTC](https://discuss.elastic.co/t/reuse-of-replicas-on-cluster-restart/13781/7 "2017-07-06T02:14:21Z")