# Need Clarification on Shards Replication **URL:** \ **Category:** Elasticsearch\ **Created:** [January 2, 2014, 5:43am UTC](https://discuss.elastic.co/t/need-clarification-on-shards-replication/15075 "2014-01-02T05:43:00Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![ananth](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@ananth](https://discuss.elastic.co/u/ananth)\ **Post date:** [January 2, 2014, 5:43am UTC](https://discuss.elastic.co/t/need-clarification-on-shards-replication/15075/1 "2014-01-02T05:43:00Z")
I have one es master and data-node and indexing documents to that (1 shard - 1 Replica), after indexing few documents (say 1 million and still indexing docs), adding one more data node to the cluster , now the shards started replicating to new node. How this replication happens ? In the mean i am still indexing new documents to that index. How will _indices.store.throttle.type: merge & indices.store.throttle.max\_bytes\_per\_sec: 50mb_ these settings react with respect to the above test scenario ? Anantha Govindarajan. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/326cfecc-b59c-4e4c-b5e9-e369e841a02e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/326cfecc-b59c-4e4c-b5e9-e369e841a02e%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 2, 2014, 8:32am UTC](https://discuss.elastic.co/t/need-clarification-on-shards-replication/15075/2 "2014-01-02T08:32:50Z")
Hey, replication is done per document (as opposed to relocation). So the document is indexed on the primary first, and if it was successful there, the document is indexed on all replicas of a shard in parallel. If that index operation on the replica(s) has returned, the index requests is returned to the client. The throttling of merges (which is a heavy I/O and CPU intensive background process) ensures, you have enough I/O performance available for index and search operations. Hope this helps... --Alex On Thu, Jan 2, 2014 at 6:43 AM, Anantha Govindarajan \< [ananthagovindarajan@gmail.com](mailto:ananthagovindarajan@gmail.com)\> wrote: > I have one es master and data-node and indexing documents to that (1 shard > > - 1 Replica), after indexing few documents (say 1 million and still > indexing docs), adding one more data node to the cluster , now the shards > started replicating to new node. How this replication happens ? In the > mean i am still indexing new documents to that index. > > How will _indices.store.throttle.type: merge > & indices.store.throttle.max\_bytes\_per\_sec: 50mb_ these settings react > with respect to the above test scenario ? > > Anantha Govindarajan. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/326cfecc-b59c-4e4c-b5e9-e369e841a02e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/326cfecc-b59c-4e4c-b5e9-e369e841a02e%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_EZ\_bb1hsVpLyW7Pt0UWM47GRU2iuQt\_mJPV0xoO5iSQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_EZ_bb1hsVpLyW7Pt0UWM47GRU2iuQt_mJPV0xoO5iSQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ananth](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@ananth](https://discuss.elastic.co/u/ananth)\ **Post date:** [January 2, 2014, 9:39am UTC](https://discuss.elastic.co/t/need-clarification-on-shards-replication/15075/3 "2014-01-02T09:39:29Z")
Hi Alex Thanks for replying. If i understand correctly normal indexing flow is, - Document is indexed in primary shard machine , then replica shard machine then return the index response to client - in case of ReplicationType.SYNC. - Document is indexed in primary shard machine , then sent it to replica machine(s) if available, and wont wait for response - in case of ReplicationType.ASYNC. But my question is not normal indexing flow. I have already indexed 1 million documents in primary shard alone , at that moment no node is available for replica. after some time adding a machine to cluster , at this point new indexing documents follows normal indexing flow (am i right ? Not sure !). But my question is how existing 1 million documents in primary shard is replicated to new machine ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dd473bdb-8603-44b2-a59c-0a8f3033ad0d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dd473bdb-8603-44b2-a59c-0a8f3033ad0d%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 2, 2014, 10:32am UTC](https://discuss.elastic.co/t/need-clarification-on-shards-replication/15075/4 "2014-01-02T10:32:25Z")
It's relocation. Segments are copied over the wire. New updates/insert/delete operations which happen in the meantime are replayed from the transaction log on the new shard. HTH -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 2 janv. 2014 à 10:39, Anantha Govindarajan [ananthagovindarajan@gmail.com](mailto:ananthagovindarajan@gmail.com) a écrit : > Hi Alex > > Thanks for replying. If i understand correctly normal indexing flow is, > > Document is indexed in primary shard machine , then replica shard machine then return the index response to client - in case of ReplicationType.SYNC. > Document is indexed in primary shard machine , then sent it to replica machine(s) if available, and wont wait for response - in case of ReplicationType.ASYNC. > But my question is not normal indexing flow. I have already indexed 1 million documents in primary shard alone , at that moment no node is available for replica. > > ## after some time adding a machine to cluster , at this point new indexing documents follows normal indexing flow (am i right ? Not sure !). But my question is how existing 1 million documents in primary shard is replicated to new machine ? > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dd473bdb-8603-44b2-a59c-0a8f3033ad0d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dd473bdb-8603-44b2-a59c-0a8f3033ad0d%40googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2DA64523-60B8-4A18-86C7-4A737FADD6B1%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/2DA64523-60B8-4A18-86C7-4A737FADD6B1%40pilato.fr). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ananth](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@ananth](https://discuss.elastic.co/u/ananth)\ **Post date:** [January 2, 2014, 11:02am UTC](https://discuss.elastic.co/t/need-clarification-on-shards-replication/15075/5 "2014-01-02T11:02:29Z")
Hi David , Thanks for your reply . Until existing(not newly created) segments are fully copied to the new machine , no indexing operation will happen on replica shard right ? rather it notes down those new indexing documents in transaction log alone ?((Correct me if i am wrong)) Once all segments are copied it replays the transaction logs . if so no new documents visible for search , till segments copying process over. is it right ? \*indices.store.throttle.type: merge & indices.store.throttle.max\_bytes\_per\_sec: 50mb \*these properties related to only lucene segment merges alone am i right ? Ananth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/051a21a8-ba9f-4401-84ce-fce31a28b5fc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/051a21a8-ba9f-4401-84ce-fce31a28b5fc%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 3, 2014, 10:52am UTC](https://discuss.elastic.co/t/need-clarification-on-shards-replication/15075/6 "2014-01-03T10:52:49Z")
Until existing(not newly created) segments are fully copied to the new machine , no indexing operation will happen on replica shard right ? rather it notes down those new indexing documents in transaction log alone ?((Correct me if i am wrong)) Once all segments are copied it replays the transaction logs . if so no new documents visible for search , till segments copying process over. is it right ? Correct. Replica shard won't be in STARTED state so it won't be searchable. indices.store.throttle.type: merge & indices.store.throttle.max\_bytes\_per\_sec: 50mb these properties related to only lucene segment merges alone am i right ? See whole definition: [https://github.com/elasticsearch/elasticsearch/issues/2041](https://github.com/elasticsearch/elasticsearch/issues/2041) HTH David -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.52c69681.643c9869.11bb1%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.52c69681.643c9869.11bb1%40MacBook-Air-de-David.local). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ananth](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@ananth](https://discuss.elastic.co/u/ananth)\ **Post date:** [January 3, 2014, 12:58pm UTC](https://discuss.elastic.co/t/need-clarification-on-shards-replication/15075/7 "2014-01-03T12:58:27Z")
Hi David Thanks for your answers. I have few questions too , _What will happen if datanode1 restarted while sending segments to datanode2_ 1. On datanode1 start-up, which will be become primary shard ? Shard in datanode1 or shard in datanode2 ? 2. If datanode1 becomes primary then how would it knows , how many amount of segments are transferred to datanode2 before restart happen? 3. if datanode2 becomes primary , what will happen to already written segments (very earlier - pending segments to be copied) ? Ananth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c79d4e4c-00df-48a0-92fd-0d5b3e48f7fc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c79d4e4c-00df-48a0-92fd-0d5b3e48f7fc%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:58am UTC](https://discuss.elastic.co/t/need-clarification-on-shards-replication/15075/8 "2017-07-06T01:58:30Z")