# How does elastic search move shards from hot to warm/warm to hot nodes? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 28, 2019, 7:18am UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005 "2019-05-28T07:18:48Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![haps\_01](https://avatars.discourse-cdn.com/v4/letter/h/ea666f/32.png) [@haps\_01](https://discuss.elastic.co/u/haps_01)\ **Post date:** [May 28, 2019, 7:18am UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/1 "2019-05-28T07:18:49Z")
Hi, I wonder, how does shard relocation happens from hot to warm nodes. I have test cluster with 2 hot, 4 warm, 2 master,1 client node. I am using multiple indices with different shard combinations(1,2,4,8,16 shards) to check how the shards are moving from one node to another. What i've seen is, as the number of shards are getting increased, it is transferring more shards(primary or replica) to target node. It would be great if someone can answer following questions . 1. Who controls the shard movement or who takes care of shard relocation, is it master or client ? 2. It relocates some given number of shards at a time, it does not relocates all at once. How does elastic decide which shard to transfer and how many shards to transfer at a given time. 3. If we relocate multiple indices from hot to warm, how does elastic decides which index to transfer first ? Thanks, ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 28, 2019, 2:21pm UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/2 "2019-05-28T14:21:33Z")
> [@haps\_01](#): > > Who controls the shard movement or who takes care of shard relocation, is it master or client ? The master. > [@haps\_01](#): > > It relocates some given number of shards at a time, it does not relocates all at once. How does elastic decide which shard to transfer and how many shards to transfer at a given time. The number of concurrent recoveries defaults to 2 incoming/outgoing (at each node) to ensure your cluster is not overloaded with recoveries. The order in which shards are relocated isn't really defined anywhere, nor is it something on which you should rely as it might change from version to version. > [@haps\_01](#): > > If we relocate multiple indices from hot to warm, how does elastic decides which index to transfer first ? It isn't really defined anywhere, nor is it something on which you should rely. Why do you want to know this, out of interest? ---
**Author:** ![haps\_01](https://avatars.discourse-cdn.com/v4/letter/h/ea666f/32.png) [@haps\_01](https://discuss.elastic.co/u/haps_01)\ **Post date:** [May 29, 2019, 7:52am UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/4 "2019-05-29T07:52:57Z")
Thanks David for the reply. 🙂 > The number of concurrent recoveries defaults to 2 incoming/outgoing (at each node) to ensure your cluster is not overloaded with recoveries. The order in which shards are relocated isn't really defined anywhere, nor is it something on which you should rely as it might change from version to version. Below is the particular snapshot of shard relocation. we could see hot-1 node is relocating more than 4 (\> 2) nodes at a particular instance. does elastic controls this according to cluster health and also can we change the default concurrent recoveries (any property/conf.) ? ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/5/b55111ff60f8a02011e4552c1cf3caabebf562f4.png) index\_16 2 p RELOCATING 1250416 530.5mb dev-es-hot-1 -\> eogqYqiTTPy11klul6ErKA dev-es-warm-3 index\_16 14 r RELOCATING 1249982 529.9mb dev-es-hot-2 -\> i0U\_2m59Q2GA5zvqfy7QzA dev-es-warm-4 index\_16 1 r RELOCATING 1248257 529.1mb dev-es-hot-1 -\> Q5OPHgH7Q\_SjdW2S20PVyA dev-es-warm-2 index\_16 12 r RELOCATING 1247503 530.5mb dev-es-hot-1 -\> eogqYqiTTPy11klul6ErKA dev-es-warm-3 index\_16 12 p RELOCATING 1247503 530.5mb dev-es-hot-2 -\> Q5OPHgH7Q\_SjdW2S20PVyA dev-es-warm-2 index\_16 4 p RELOCATING 1252275 530.8mb dev-es-hot-1 -\> i0U\_2m59Q2GA5zvqfy7QzA dev-es-warm-4 > It isn't really defined anywhere, nor is it something on which you should rely. Why do you want to know this, out of interest? i was just wondering if there is any algorithm/strategy which takes care of this movement. I was going through the below blog: > **[Every Shard deserves a home.](https://www.elastic.co/blog/every-shard-deserves-a-home)** > > Every Shard deserves a home. An explanation of Shard allocation within Elasticsearch. According to this blog all the new indexing operations are sent to new primary node(target). If we perform any query during this relocation, how does elastic make sure it gives correct result because old primary(source) does not contain the new writes ? In shard relocation process when the primary node completes the movement, does replica always gets copied from the primary node instead of old replica node(source) ? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 29, 2019, 10:33am UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/5 "2019-05-29T10:33:12Z")
> [@haps\_01](#): > > does elastic controls this according to cluster health and also can we change the default concurrent recoveries (any property/conf.) ? It doesn't really depend on cluster health, no, but there [are settings to adjust this](https://www.elastic.co/guide/en/elasticsearch/reference/current/shards-allocation.html). To be clear, if you set these much higher than the defaults then you are putting the stability of your cluster at risk. > [@haps\_01](#): > > i was just wondering if there is any algorithm/strategy which takes care of this movement. Not really, no, Elasticsearch just keeps on moving shards until they're all in the right places. > [@haps\_01](#): > > how does elastic make sure it gives correct result because old primary(source) does not contain the new writes ? Elasticsearch keeps on writing to the old primary until the new primary is ready to take over. > [@haps\_01](#): > > when the primary node completes the movement, does replica always gets copied from the primary node instead of old replica node(source) ? I don't really understand. The replicas are already up-to-date so there's no need to copy them from anywhere. ---
**Author:** ![haps\_01](https://avatars.discourse-cdn.com/v4/letter/h/ea666f/32.png) [@haps\_01](https://discuss.elastic.co/u/haps_01)\ **Post date:** [May 29, 2019, 12:47pm UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/6 "2019-05-29T12:47:05Z")
> Elasticsearch keeps on writing to the old primary until the new primary is ready to take over. does it mean when elastic relocates any index from hot to warm nodes, all new indexing operations sent to both primary(source and target node) ? Quote from the blog: "In 2.x/5.x, we're better. As soon as we start relocation, primary will start to send all indexing operations to the new primary(node5)." > I don't really understand. The replicas are already up-to-date so there's no need to copy them from anywhere. When elastic relocates any index from hot to warm node, it will first move primary node to hot to warm and then it relocates replica. My doubt was if replica shard also get copied from hot to warm node or it will be copied from the primary node ? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 29, 2019, 12:55pm UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/7 "2019-05-29T12:55:46Z")
> [@haps\_01](#): > > does it mean when elastic relocates any index from hot to warm nodes, all new indexing operations sent to both primary(source and target node) ? Yes, pretty much. The "target" copy is more like a replica than a primary for most of the relocation process, and like all other replicas it receives all indexing operations. However, this question is a bit strange because you wouldn't normally be indexing into an index that's being moved to a warm node. > [@haps\_01](#): > > My doubt was if replica shard also get copied from hot to warm node or it will be copied from the primary node ? I see. Yes, a new replica is always built by copying information from the current primary wherever that might be. ---
**Author:** ![haps\_01](https://avatars.discourse-cdn.com/v4/letter/h/ea666f/32.png) [@haps\_01](https://discuss.elastic.co/u/haps_01)\ **Post date:** [June 3, 2019, 1:30pm UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/8 "2019-06-03T13:30:30Z")
Hi David, One more doubt. How does primary and replica node get transferred ? is it parallel or sequential. We have seen different result for different shard combinations. We are using below command to capture the time taken in shard movement. **GET /index\_3/\_recovery?human&detailed=true** Output of this command contains start\_time and stop\_time which tells the exact timing of shard movement. Below is the consolidated data for some sample indices. | index | number of shards | time taken per shard | start\_time | stop\_time | total time for index | | --- | --- | --- | --- | --- | --- | | index\_1 | 1 | 6.6~6.7m | 2019-06-03T11:37:44.679Z | 2019-06-03T11:44:32.222Z | 0:06:48 | | index\_2 | 2 | 2.2~3.4m | 2019-06-03T11:50:04.463Z | 2019-06-03T11:59:09.779Z | 0:09:05 | | index\_3 | 4 | 1.6~1.7m | 2019-06-03T11:59:17.880Z | 2019-06-03T12:06:15.461Z | 0:06:58 | | index\_4 | 8 | 34.3s~1.5m | 2019-06-03T12:22:02.480Z | 2019-06-03T12:27:13.280Z | 0:05:11 | | index\_16 | 16 | 16.4s~1.7m | 2019-06-03T12:27:43.571Z | 2019-06-03T12:37:52.864Z | 0:10:09 | If we check the start and stop time per shard, we can see some of the shards are getting relocated in parallel while some are sequential. It would be great help of you can explain this behavior. We have cluster with 2 hot node and 4 warm node. We are relocating a index with 2 shard and 1 replica per shard(total 4 shards). After the relocation , when we check \_recovery command, It shows source as hot and target as warm node for some shards while source as warm and target as warm for some. e.g. "source": { "id": "eogqYqiTTPy11klul6ErKA", "name": "dev-es-warm-3" }, "target": { "id": "Q5OPHgH7Q\_SjdW2S20PVyA", "name": "dev-es-warm-2" } "source": { "id": "IoyUloOfR1acbnkE2Bv1-w", "name": "dev-es-hot-1" }, "target": { "id": "i0U\_2m59Q2GA5zvqfy7QzA", "name": "dev-es-warm-4" } I am little confused about the replica shard movement. is it parallel or sequential. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 3, 2019, 2:13pm UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/9 "2019-06-03T14:13:20Z")
> [@haps\_01](#): > > I am little confused about the replica shard movement. is it parallel or sequential. It's both. Recoveries happen in parallel, up to a limit that prevents the cluster from spending too many resources on recoveries. Once the limit is reached, further recoveries have to wait for the ongoing ones to finish. ---
**Author:** ![haps\_01](https://avatars.discourse-cdn.com/v4/letter/h/ea666f/32.png) [@haps\_01](https://discuss.elastic.co/u/haps_01)\ **Post date:** [June 3, 2019, 4:19pm UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/10 "2019-06-03T16:19:54Z")
> It's both. Recoveries happen in parallel, up to a limit that prevents the cluster from spending too many resources on recoveries. Once the limit is reached, further recoveries have to wait for the ongoing ones to finish. Okay. Also one more thing, is it always guaranteed that primary shard will relocate first ? or replica node can also move from hot to warm node. ? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 3, 2019, 4:43pm UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/11 "2019-06-03T16:43:11Z")
> [@haps\_01](#): > > is it always guaranteed that primary shard will relocate first ? No, I don't think there is any such guarantee. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 1, 2019, 4:43pm UTC](https://discuss.elastic.co/t/how-does-elastic-search-move-shards-from-hot-to-warm-warm-to-hot-nodes/183005/12 "2019-07-01T16:43:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.