# Rolling Restart -- Local Replicas do not reuse local data (2.4.2) **URL:** \ **Category:** Elasticsearch\ **Created:** [June 20, 2017, 9:02pm UTC](https://discuss.elastic.co/t/rolling-restart-local-replicas-do-not-reuse-local-data-2-4-2/90162 "2017-06-20T21:02:42Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![promiseofcake](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/promiseofcake/32/18365_2.png) [@promiseofcake](https://discuss.elastic.co/u/promiseofcake)\ **Post date:** [June 20, 2017, 9:02pm UTC](https://discuss.elastic.co/t/rolling-restart-local-replicas-do-not-reuse-local-data-2-4-2/90162/1 "2017-06-20T21:02:43Z")
We are in the process of doing some rolling restarts on our Elasticsearch data nodes and running into issues relating to shard recovery. When I performed an upgrade from 2.3.x to 2.4.2 we followed these steps and did not have any such problems: I will begin with some questions, and then outline what we are doing below: ## Questions - How can we reliably force previous primaries, and current replicas to recover from a restarted node as opposed to from remote nodes? - How can we get logs on why the decisions were made to replicate from remote primaries as opposed to local replicas? ## 2.3.x to 2.4.2 Upgrade (Smooth sailing) We performed the following for an effortless 2.3 - 2.4 upgrade last year. **Stop Indexing** **Disable shard Allocation** on es-master: ``` curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d '{ { "transient": { "cluster.routing.allocation.enable": "none" } }' ``` **Perform Synced Flush (and ensure results are success)** on es-master: ``` curl -XPOST 'localhost:9200/_all/_flush/synced?pretty' ``` **Restart the node (using pseudocode)** on es-data-node: ``` stop elasticsearch start elasticsearch ``` **Wait for node to return to cluster** on es-master: ``` curl localhost:9200/_cat/nodes?v ``` Once we are sure the node is back, re-enable shard-allocation **Re-enable Shard Allocation** on es-master: ``` curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d' { "transient": { "cluster.routing.allocation.enable": "all" } }' ``` **Observe that shards allocate via local data on that node** And everything worked great. We had a cluster with 24 data nodes, 2 client nodes and 3 master nodes. Within that cluster 6 - 8 indexes with shard counts between 5 - 24 with a replication factor of 1. ## 2.4.2 Rolling Restart (troubled waters) Now our shards are a on the higher end of the recommended 30GB size, so when we have attempted to do another rolling-restart of our data nodes (no version update), we have not seen the same behavior. 1, 2, 3, 4 are all done the same, but when 5. is executed (re-enable shard allocation), the nodes are being assigned to new nodes, or with forced assignment, copying their data from new nodes rather than the local dataset. Here is a list of things we have tried to do to rectify the situation: **Set node\_left.delayed\_timeout to 10mins** We were hoping that just this option would do it for us, but we aren't seeing that working either. ``` curl -XPUT "localhost:9200/_all/_settings" -d ' { "settings": { "index.unassigned.node_left.delayed_timeout": "10m" } }' ``` **Force all allocation off / on** ``` curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d ' { "transient": { "cluster.routing.rebalance.enable": "NONE", "cluster.routing.allocation.enable": "none", "cluster.routing.allocation.disable_allocation": "true" } }' curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d ' { "transient": { "cluster.routing.rebalance.enable": "ALL", "cluster.routing.allocation.enable": "all", "cluster.routing.allocation.disable_allocation": "false" } }' ``` **Force Allocate Shards to the same node** ``` curl -XPOST 'localhost:9200/_cluster/reroute' -d ' { "commands": [ {"allocate": {"index": "pix_v1", "shard": 9, "node": "foo-bar-es-data-i-713814"}}, {"allocate": {"index": "pix_v1", "shard": 4, "node": "foo-bar-es-data-i-713814"}}, {"allocate": {"index": "pix_v2", "shard": 22, "node": "foo-bar-es-data-i-713814"}}, {"allocate": {"index": "pix_v2", "shard": 15, "node": "foo-bar-es-data-i-713814"}}, {"allocate": {"index": "people_v0", "shard": 4, "node": "foo-bar-es-data-i-713814"}} ] }' ``` Here we just see the _foo-bar-es-data-i-713814_ node being hit with high IO when attempting to read the data from other member node primaries. It is ignoring the local copy. **Attempt Different Combinations of the Disable Options** We have attempted to keep cluster rebalancing off and just allocation on, and vice versa. Current Settings: ``` { "transient": { "logger": { "indices": { "recovery": "DEBUG" }, "cluster": { "service": "DEBUG", "routing": "DEBUG" }, }, "indices": { "recovery": { "max_bytes_per_sec": "40mb", "concurrent_streams": "3", "concurrent_small_file_streams": "2" } }, "cluster": { "routing": { "allocation": { "enable": "all", "node_concurrent_recoveries": "2", "cluster_concurrent_rebalance": "2", "disable_allocation": "false" }, "rebalance": { "enable": "ALL" } } } }, "persistent": { "indices": { "recovery": { "max_bytes_per_sec": "200mb", "concurrent_streams": "5" } }, "cluster": { "routing": { "allocation": { "node_concurrent_recoveries": "5" } } } } } ``` Any guidance would be helpful. Thank you. ---
**Author:** ![bleskes](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@bleskes](https://discuss.elastic.co/u/bleskes)\ **Post date:** [June 23, 2017, 6:57am UTC](https://discuss.elastic.co/t/rolling-restart-local-replicas-do-not-reuse-local-data-2-4-2/90162/2 "2017-06-23T06:57:31Z")
Two questions that quickly come to mind: 1. Did you stop indexing? indexing violates the synced flush markers 2. Did you check that the output of the synced flushing indicated it was successful? ---
**Author:** ![promiseofcake](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/promiseofcake/32/18365_2.png) [@promiseofcake](https://discuss.elastic.co/u/promiseofcake)\ **Post date:** [June 23, 2017, 3:28pm UTC](https://discuss.elastic.co/t/rolling-restart-local-replicas-do-not-reuse-local-data-2-4-2/90162/3 "2017-06-23T15:28:07Z")
@bleskes, Yes we stopped our ingress pipeline and observed our Elasticsearch metrics to ensure no index operations were being propagated to the cluster. Also on the synced flush yes, we saw something similar to: ``` { "_shards" : { "total" : 135, "successful" : 135, "failed" : 0 }, "blomp_v0" : { "total" : 10, "successful" : 10, "failed" : 0 }, "baz_v1" : { "total" : 48, "successful" : 48, "failed" : 0 }, ".kibana" : { "total" : 2, "successful" : 2, "failed" : 0 }, "baz_v2" : { "total" : 48, "successful" : 48, "failed" : 0 }, "boop" : { "total" : 2, "successful" : 2, "failed" : 0 }, "foo_v0" : { "total" : 10, "successful" : 10, "failed" : 0 }, "bar_v0" : { "total" : 10, "successful" : 10, "failed" : 0 } } ``` I was originally thinking that some rogue index operations could have been causing the primaries to become out of sync but I am positive we have stopped all incoming index jobs. Something interesting from last time is that we have two (2) types of data-nodes in the cluster now, we have `a` and `b`. `a` is where most of the data lives, but we run some CPU intensive queries on `b` nodes so we have defined rules to allocate certain indices to certain nodes. Could that be confusing the balancer in some way? ---
**Author:** ![bleskes](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@bleskes](https://discuss.elastic.co/u/bleskes)\ **Post date:** [June 23, 2017, 4:13pm UTC](https://discuss.elastic.co/t/rolling-restart-local-replicas-do-not-reuse-local-data-2-4-2/90162/4 "2017-06-23T16:13:01Z")
If the cluster was stable before than those allocations rules should not confuse the balancer. I also understand the you tried force allocating to _foo-bar-es-data-i-713814_, a node you know has data and still the recovery takes long. This means that the balancer made the right allocation decision but that the recovery didn't reuse files. Here is what I suggest you do - choose a single shard on a node you want to monitor. When your index is green, stop indexing and issue synced flush. Use the procedure described [here](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/indices-synced-flush.html) to see that the synced flush markers are put on both shard copies (and are the same). Restart the node hosting the shard and see how it goes. You can set the `indices.recovery` log to `TRACE` before the restart and it will dump a _lot_ of information about the recovery and why it does what it does. ---
**Author:** ![promiseofcake](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/promiseofcake/32/18365_2.png) [@promiseofcake](https://discuss.elastic.co/u/promiseofcake)\ **Post date:** [June 23, 2017, 4:41pm UTC](https://discuss.elastic.co/t/rolling-restart-local-replicas-do-not-reuse-local-data-2-4-2/90162/5 "2017-06-23T16:41:01Z")
> [@bleskes](#): > > If the cluster was stable before than those allocations rules should confuse the balancer. I also understand the you tried force allocating to foo-bar-es-data-i-713814, a node you know has data and still the recovery takes long. This means that the balancer made the right allocation decision but that the recovery didn't reuse files. It's hard to say that was the only change because we also upgraded from 2.3 to 2.4.2 at that time so it could be something introduced in the version upgrade as well. Thank you for that suggestion @bleskes. I will give that a try and report back. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 21, 2017, 4:41pm UTC](https://discuss.elastic.co/t/rolling-restart-local-replicas-do-not-reuse-local-data-2-4-2/90162/6 "2017-07-21T16:41:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.