# Shards hot relocation **URL:** \ **Category:** Elasticsearch\ **Created:** [May 14, 2012, 9:12pm UTC](https://discuss.elastic.co/t/shards-hot-relocation/7691 "2012-05-14T21:12:08Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Bing\_Hua](https://avatars.discourse-cdn.com/v4/letter/b/e99b99/32.png) [@Bing\_Hua](https://discuss.elastic.co/u/Bing_Hua)\ **Post date:** [May 14, 2012, 9:12pm UTC](https://discuss.elastic.co/t/shards-hot-relocation/7691/1 "2012-05-14T21:12:08Z")
A general question: Say we have a cluster running and constantly getting index requests coming in. When a new node is brought up, some shards are re-allocated to this node. What is happening to the source nodes of the shards? Are they still processing new index requests during shard relocation? How do they transfer indices while indices are changing? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 15, 2012, 8:54pm UTC](https://discuss.elastic.co/t/shards-hot-relocation/7691/2 "2012-05-15T20:54:21Z")
Yes, they are still processing the indexing requests until the relocation is done. Its done in several stages (the relocation, or recovery for that matter). On Tue, May 15, 2012 at 12:12 AM, Bing Hua [bh349@cornell.edu](mailto:bh349@cornell.edu) wrote: > A general question: Say we have a cluster running and constantly getting > index requests coming in. When a new node is brought up, some shards are > re-allocated to this node. What is happening to the source nodes of the > shards? Are they still processing new index requests during shard > relocation? How do they transfer indices while indices are changing? ---
**Author:** ![Bing](https://avatars.discourse-cdn.com/v4/letter/b/839c29/32.png) [@Bing](https://discuss.elastic.co/u/Bing)\ **Post date:** [May 17, 2012, 7:53pm UTC](https://discuss.elastic.co/t/shards-hot-relocation/7691/3 "2012-05-17T19:53:49Z")
Thanks kimchy. It's good to know they still process indexing but still I'm curious on how do they 'process' indexing to a 'moving' shard. Is it achieved by using transaction log? So the source node marks the happening of shard relocation, does the index transfer while recording incoming requests but not merging them into the index. After the transfer is done, it sends the part of trans log to the new node to have it process indexing? Bing On May 15, 3:54 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Yes, they are still processing the indexing requests until the relocation > is done. Its done in several stages (the relocation, or recovery for that > matter). > > On Tue, May 15, 2012 at 12:12 AM, Bing Hua [bh...@cornell.edu](mailto:bh...@cornell.edu) wrote: > > > A general question: Say we have a cluster running and constantly getting > > index requests coming in. When a new node is brought up, some shards are > > re-allocated to this node. What is happening to the source nodes of the > > shards? Are they still processing new index requests during shard > > relocation? How do they transfer indices while indices are changing? ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [May 18, 2012, 3:12am UTC](https://discuss.elastic.co/t/shards-hot-relocation/7691/4 "2012-05-18T03:12:46Z")
Hi, I think this question was also answered during Shay's presentation on last #BBUZZ. Here is the video: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Here are the slides: > **[elasticsearch-bbuzz2011.pdf](https://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/elasticsearch-bbuzz2011.pdf)** > > 2.25 MB More specifically, in the video, skip to [37:25] (although it can be hard to make vimeo skip to it), it goes like: _Q: I am just wondering how to hot failover works, do you use push for that? Do you replay the transaction log or... A: yea, so you mean the hot relocation? Q: yes A: When the relocation happens, it gets smart in Lucene itself by making sure that we do not delete the segment files of Lucene, and we start to transfer these segment files, and we disable flushing, we do not call commit anymore on Lucene and we only store the changes into transaction log. Once that phase is done (ie. we copied over all the index files), we start replaying the transaction log into to replica and then we do the switch and tell it, ok, there is the new one._ \* \* _HTH_ \* \* _Regards,_ _Lukas_ On Thu, May 17, 2012 at 9:53 PM, Bing [jasoninelmstreet@gmail.com](mailto:jasoninelmstreet@gmail.com) wrote: > Thanks kimchy. It's good to know they still process indexing but still > I'm curious on how do they 'process' indexing to a 'moving' shard. Is > it achieved by using transaction log? So the source node marks the > happening of shard relocation, does the index transfer while recording > incoming requests but not merging them into the index. After the > transfer is done, it sends the part of trans log to the new node to > have it process indexing? > > Bing > > On May 15, 3:54 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Yes, they are still processing the indexing requests until the relocation > > is done. Its done in several stages (the relocation, or recovery for that > > matter). > > > > On Tue, May 15, 2012 at 12:12 AM, Bing Hua [bh...@cornell.edu](mailto:bh...@cornell.edu) wrote: > > > > > A general question: Say we have a cluster running and constantly > > > getting > > > index requests coming in. When a new node is brought up, some shards > > > are > > > re-allocated to this node. What is happening to the source nodes of the > > > shards? Are they still processing new index requests during shard > > > relocation? How do they transfer indices while indices are changing? ---
**Author:** ![Bing](https://avatars.discourse-cdn.com/v4/letter/b/839c29/32.png) [@Bing](https://discuss.elastic.co/u/Bing)\ **Post date:** [May 18, 2012, 2:41pm UTC](https://discuss.elastic.co/t/shards-hot-relocation/7691/5 "2012-05-18T14:41:59Z")
Thanks Lukas, so looks like my point was pretty much the case how it is implemented, right? The source node is accepting new index requests but not actually 'processing' them. After a while the list of pending requests are sent to new node and processed by the new node. Bing On May 17, 10:12 pm, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > Hi, > > I think this question was also answered during Shay's presentation on last > #BBUZZ. > > Here is the video:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2011/08/09/road-to-a-distributed-)... > Here are the slides:[http://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/el](http://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/el)... > > More specifically, in the video, skip to [37:25] (although it can be hard > to make vimeo skip to it), it goes like: > > _Q: I am just wondering how to hot failover works, do you use push for > that? Do you replay the transaction log or... > A: yea, so you mean the hot relocation? > Q: yes > A: When the relocation happens, it gets smart in Lucene itself by making > sure that we do not delete the segment files of Lucene, and we start to > transfer these segment files, and we disable flushing, we do not call > commit anymore on Lucene and we only store the changes into transaction > log. Once that phase is done (ie. we copied over all the index files), we > start replaying the transaction log into to replica and then we do the > switch and tell it, ok, there is the new one._ > \* > \* > _HTH_ > \* > \* > _Regards,_ > _Lukas_ > > On Thu, May 17, 2012 at 9:53 PM, Bing [jasoninelmstr...@gmail.com](mailto:jasoninelmstr...@gmail.com) wrote: > > > Thanks kimchy. It's good to know they still process indexing but still > > I'm curious on how do they 'process' indexing to a 'moving' shard. Is > > it achieved by using transaction log? So the source node marks the > > happening of shard relocation, does the index transfer while recording > > incoming requests but not merging them into the index. After the > > transfer is done, it sends the part of trans log to the new node to > > have it process indexing? > > > Bing > > > On May 15, 3:54 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > Yes, they are still processing the indexing requests until the relocation > > > is done. Its done in several stages (the relocation, or recovery for that > > > matter). > > > > On Tue, May 15, 2012 at 12:12 AM, Bing Hua [bh...@cornell.edu](mailto:bh...@cornell.edu) wrote: > > > > > > > A general question: Say we have a cluster running and constantly > > > > getting > > > > index requests coming in. When a new node is brought up, some shards > > > > are > > > > re-allocated to this node. What is happening to the source nodes of the > > > > shards? Are they still processing new index requests during shard > > > > relocation? How do they transfer indices while indices are changing? ---
**Author:** ![Bing](https://avatars.discourse-cdn.com/v4/letter/b/839c29/32.png) [@Bing](https://discuss.elastic.co/u/Bing)\ **Post date:** [May 18, 2012, 2:55pm UTC](https://discuss.elastic.co/t/shards-hot-relocation/7691/6 "2012-05-18T14:55:14Z")
"we only store the changes into transaction log." So it's not requests in the trans log but actual changes? Is there an example of how this piece of trans log look like? Bing On May 18, 9:41 am, Bing [jasoninelmstr...@gmail.com](mailto:jasoninelmstr...@gmail.com) wrote: > Thanks Lukas, so looks like my point was pretty much the case how it > is implemented, right? > The source node is accepting new index requests but not actually > 'processing' them. After a while the list of pending requests are sent > to new node and processed by the new node. > > Bing > > On May 17, 10:12 pm, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > > > Hi, > > > I think this question was also answered during Shay's presentation on last > > #BBUZZ. > > > Here is the video:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2011/08/09/road-to-a-distributed-)... > > Here are the slides:[http://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/el](http://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/el)... > > > More specifically, in the video, skip to [37:25] (although it can be hard > > to make vimeo skip to it), it goes like: > > > _Q: I am just wondering how to hot failover works, do you use push for > > that? Do you replay the transaction log or... > > A: yea, so you mean the hot relocation? > > Q: yes > > A: When the relocation happens, it gets smart in Lucene itself by making > > sure that we do not delete the segment files of Lucene, and we start to > > transfer these segment files, and we disable flushing, we do not call > > commit anymore on Lucene and we only store the changes into transaction > > log. Once that phase is done (ie. we copied over all the index files), we > > start replaying the transaction log into to replica and then we do the > > switch and tell it, ok, there is the new one._ > > \* > > \* > > _HTH_ > > \* > > \* > > _Regards,_ > > _Lukas_ > > > On Thu, May 17, 2012 at 9:53 PM, Bing [jasoninelmstr...@gmail.com](mailto:jasoninelmstr...@gmail.com) wrote: > > > > > Thanks kimchy. It's good to know they still process indexing but still > > > I'm curious on how do they 'process' indexing to a 'moving' shard. Is > > > it achieved by using transaction log? So the source node marks the > > > happening of shard relocation, does the index transfer while recording > > > incoming requests but not merging them into the index. After the > > > transfer is done, it sends the part of trans log to the new node to > > > have it process indexing? > > > > Bing > > > > On May 15, 3:54 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > > > Yes, they are still processing the indexing requests until the relocation > > > > is done. Its done in several stages (the relocation, or recovery for that > > > > matter). > > > > > On Tue, May 15, 2012 at 12:12 AM, Bing Hua [bh...@cornell.edu](mailto:bh...@cornell.edu) wrote: > > > > > > > > > A general question: Say we have a cluster running and constantly > > > > > getting > > > > > index requests coming in. When a new node is brought up, some shards > > > > > are > > > > > re-allocated to this node. What is happening to the source nodes of the > > > > > shards? Are they still processing new index requests during shard > > > > > relocation? How do they transfer indices while indices are changing? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 20, 2012, 8:25pm UTC](https://discuss.elastic.co/t/shards-hot-relocation/7691/7 "2012-05-20T20:25:28Z")
The data stored in the transaction log is specific ot each operation, for example, when indexing, the source document is stored there, as well as additional metadata. On Fri, May 18, 2012 at 4:55 PM, Bing [jasoninelmstreet@gmail.com](mailto:jasoninelmstreet@gmail.com) wrote: > "we only store the changes into transaction log." > So it's not requests in the trans log but actual changes? Is there an > example of how this piece of trans log look like? > > Bing > > On May 18, 9:41 am, Bing [jasoninelmstr...@gmail.com](mailto:jasoninelmstr...@gmail.com) wrote: > > > Thanks Lukas, so looks like my point was pretty much the case how it > > is implemented, right? > > The source node is accepting new index requests but not actually > > 'processing' them. After a while the list of pending requests are sent > > to new node and processed by the new node. > > > > Bing > > > > On May 17, 10:12 pm, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > > > > > Hi, > > > > > I think this question was also answered during Shay's presentation on > > > last > > > #BBUZZ. > > > > > Here is the video: > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2011/08/09/road-to-a-distributed-)... > > > Here are the slides: > > > [http://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/el](http://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/el)... > > > > > More specifically, in the video, skip to [37:25] (although it can be > > > hard > > > to make vimeo skip to it), it goes like: > > > > > _Q: I am just wondering how to hot failover works, do you use push for > > > that? Do you replay the transaction log or... > > > A: yea, so you mean the hot relocation? > > > Q: yes > > > A: When the relocation happens, it gets smart in Lucene itself by > > > making > > > sure that we do not delete the segment files of Lucene, and we start to > > > transfer these segment files, and we disable flushing, we do not call > > > commit anymore on Lucene and we only store the changes into transaction > > > log. Once that phase is done (ie. we copied over all the index files), > > > we > > > start replaying the transaction log into to replica and then we do the > > > switch and tell it, ok, there is the new one._ > > > \* > > > \* > > > _HTH_ > > > \* > > > \* > > > _Regards,_ > > > _Lukas_ > > > > > On Thu, May 17, 2012 at 9:53 PM, Bing [jasoninelmstr...@gmail.com](mailto:jasoninelmstr...@gmail.com) > > > wrote: > > > > > > > Thanks kimchy. It's good to know they still process indexing but > > > > still > > > > I'm curious on how do they 'process' indexing to a 'moving' shard. Is > > > > it achieved by using transaction log? So the source node marks the > > > > happening of shard relocation, does the index transfer while > > > > recording > > > > incoming requests but not merging them into the index. After the > > > > transfer is done, it sends the part of trans log to the new node to > > > > have it process indexing? > > > > > > Bing > > > > > > On May 15, 3:54 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > > > > > Yes, they are still processing the indexing requests until the > > > > > relocation > > > > > is done. Its done in several stages (the relocation, or recovery > > > > > for that > > > > > matter). > > > > > > > On Tue, May 15, 2012 at 12:12 AM, Bing Hua [bh...@cornell.edu](mailto:bh...@cornell.edu) > > > > > wrote: > > > > > > > > > > > A general question: Say we have a cluster running and constantly > > > > > > getting > > > > > > index requests coming in. When a new node is brought up, some > > > > > > shards > > > > > > are > > > > > > re-allocated to this node. What is happening to the source nodes > > > > > > of the > > > > > > shards? Are they still processing new index requests during shard > > > > > > relocation? How do they transfer indices while indices are > > > > > > changing? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:27am UTC](https://discuss.elastic.co/t/shards-hot-relocation/7691/8 "2017-07-06T03:27:58Z")