# Why ES node starts recovering all the data from other nodes after reboot? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 21, 2014, 2:48am UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881 "2014-11-21T02:48:56Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Konstantin\_Erman](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@Konstantin\_Erman](https://discuss.elastic.co/u/Konstantin_Erman)\ **Post date:** [November 21, 2014, 2:48am UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/1 "2014-11-21T02:48:56Z")
I work on an experimental cluster of ES nodes running on Windows Server machines. Once in a while we have a need to reboot machines. The initial state - cluster is green and well balanced. One machine is gracefully taken offline and then after necessary service is performed it comes back online. All the hardware and file system content is intact. As soon as ES service starts on that machine, it assumes that there is no usable data locally and recovers as much data as it deems necessary for balancing from other nodes. This behavior puzzles me, because most of the data shards stored on that machine file system can be reused as they are. Cluster stores logs, so all indices except those for the current day never ever change until they get deleted. Can't ES node detect that it has perfect copies of some (actually most) of the shards and instead of copying them over just mark them as up to date? I suspect I don't know about some step to enable this behavior and I'm looking to enable it. Any advice? Thank you! Konstantin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 21, 2014, 2:51am UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/2 "2014-11-21T02:51:58Z")
You should disable allocation before you reboot, that will save a lot of shard shuffling - > **[Elasticsearch Platform โ€” Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On 21 November 2014 13:48, Konstantin Erman [konste@gmail.com](mailto:konste@gmail.com) wrote: > I work on an experimental cluster of ES nodes running on Windows Server > machines. Once in a while we have a need to reboot machines. The initial > state - cluster is green and well balanced. One machine is gracefully taken > offline and then after necessary service is performed it comes back online. > All the hardware and file system content is intact. As soon as ES service > starts on that machine, it assumes that there is no usable data locally and > recovers as much data as it deems necessary for balancing from other nodes. > > This behavior puzzles me, because most of the data shards stored on that > machine file system can be reused as they are. Cluster stores logs, so all > indices except those for the current day never ever change until they get > deleted. Can't ES node detect that it has perfect copies of some (actually > most) of the shards and instead of copying them over just mark them as up > to date? > > I suspect I don't know about some step to enable this behavior and I'm > looking to enable it. Any advice? > > Thank you! > Konstantin > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmc\_rMFzRUUrJSMJ9bY16tz-dZ8eSeUZobC7XaxWZTRPg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmc_rMFzRUUrJSMJ9bY16tz-dZ8eSeUZobC7XaxWZTRPg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yves\_Dorfsman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yves_dorfsman/32/1106_2.png) [@Yves\_Dorfsman](https://discuss.elastic.co/u/Yves_Dorfsman)\ **Post date:** [November 21, 2014, 3:51am UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/3 "2014-11-21T03:51:52Z")
If you do disable allocation before you reboot a node and a client writes to a shard that had a replica on that node, does the entire replica gets copied when the node come up? Or does it get just updated? On Thursday, 20 November 2014 19:52:26 UTC-7, Mark Walkom wrote: > You should disable allocation before you reboot, that will save a lot of > shard shuffling - > [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#rolling-upgrades) > > On 21 November 2014 13:48, Konstantin Erman \<[kon...@gmail.com](mailto:kon...@gmail.com) > \\> wrote: > > > I work on an experimental cluster of ES nodes running on Windows Server > > machines. Once in a while we have a need to reboot machines. The initial > > state - cluster is green and well balanced. One machine is gracefully taken > > offline and then after necessary service is performed it comes back online. > > All the hardware and file system content is intact. As soon as ES service > > starts on that machine, it assumes that there is no usable data locally and > > recovers as much data as it deems necessary for balancing from other nodes. > > > > This behavior puzzles me, because most of the data shards stored on that > > machine file system can be reused as they are. Cluster stores logs, so all > > indices except those for the current day never ever change until they get > > deleted. Can't ES node detect that it has perfect copies of some (actually > > most) of the shards and instead of copying them over just mark them as up > > to date? > > > > I suspect I don't know about some step to enable this behavior and I'm > > looking to enable it. Any advice? > > > > Thank you! > > Konstantin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 21, 2014, 4:10am UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/4 "2014-11-21T04:10:35Z")
It will enter recovery where it syncs at the segment level from the current primary, then the translog gets shipped over and (re)played, which brings it all up to date. On 21 November 2014 14:51, Yves Dorfsman [yves@zioup.com](mailto:yves@zioup.com) wrote: > If you do disable allocation before you reboot a node and a client writes > to a shard that had a replica on that node, does the entire replica gets > copied when the node come up? Or does it get just updated? > > On Thursday, 20 November 2014 19:52:26 UTC-7, Mark Walkom wrote: > > > You should disable allocation before you reboot, that will save a lot of > > shard shuffling - [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > reference/current/setup-upgrade.html#rolling-upgrades > > > > On 21 November 2014 13:48, Konstantin Erman [kon...@gmail.com](mailto:kon...@gmail.com) wrote: > > > > > I work on an experimental cluster of ES nodes running on Windows Server > > > machines. Once in a while we have a need to reboot machines. The initial > > > state - cluster is green and well balanced. One machine is gracefully taken > > > offline and then after necessary service is performed it comes back online. > > > All the hardware and file system content is intact. As soon as ES service > > > starts on that machine, it assumes that there is no usable data locally and > > > recovers as much data as it deems necessary for balancing from other nodes. > > > > > > This behavior puzzles me, because most of the data shards stored on that > > > machine file system can be reused as they are. Cluster stores logs, so all > > > indices except those for the current day never ever change until they get > > > deleted. Can't ES node detect that it has perfect copies of some (actually > > > most) of the shards and instead of copying them over just mark them as up > > > to date? > > > > > > I suspect I don't know about some step to enable this behavior and I'm > > > looking to enable it. Any advice? > > > > > > Thank you! > > > Konstantin > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj\_yKT6d8\_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj_yKT6d8_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [November 21, 2014, 5:02am UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/5 "2014-11-21T05:02:16Z")
The thing is that this is a disk level operation. It pretty much rsyncs the files from the current master shard to the node when it comes back online. This would be OK if the replica shards matched the master but that is only normally the case if the shard was moved to the node after it was mostly complete and then you've had only a few writes. Normally shards don't match each other because the way the index is maintained is nondeterministic. The translog replay is only used as a catch up after the rsync-like step. This is something that is being worked on. Its certainly my biggest complaint about elasticsearch but I'm confident that it'll get better. Nik On Nov 20, 2014 11:11 PM, "Mark Walkom" [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > It will enter recovery where it syncs at the segment level from the > current primary, then the translog gets shipped over and (re)played, which > brings it all up to date. > > On 21 November 2014 14:51, Yves Dorfsman [yves@zioup.com](mailto:yves@zioup.com) wrote: > > > If you do disable allocation before you reboot a node and a client writes > > to a shard that had a replica on that node, does the entire replica gets > > copied when the node come up? Or does it get just updated? > > > > On Thursday, 20 November 2014 19:52:26 UTC-7, Mark Walkom wrote: > > > > > You should disable allocation before you reboot, that will save a lot of > > > shard shuffling - [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > > reference/current/setup-upgrade.html#rolling-upgrades > > > > > > On 21 November 2014 13:48, Konstantin Erman [kon...@gmail.com](mailto:kon...@gmail.com) wrote: > > > > > > > I work on an experimental cluster of ES nodes running on Windows Server > > > > machines. Once in a while we have a need to reboot machines. The initial > > > > state - cluster is green and well balanced. One machine is gracefully taken > > > > offline and then after necessary service is performed it comes back online. > > > > All the hardware and file system content is intact. As soon as ES service > > > > starts on that machine, it assumes that there is no usable data locally and > > > > recovers as much data as it deems necessary for balancing from other nodes. > > > > > > > > This behavior puzzles me, because most of the data shards stored on > > > > that machine file system can be reused as they are. Cluster stores logs, so > > > > all indices except those for the current day never ever change until they > > > > get deleted. Can't ES node detect that it has perfect copies of some > > > > (actually most) of the shards and instead of copying them over just mark > > > > them as up to date? > > > > > > > > I suspect I don't know about some step to enable this behavior and I'm > > > > looking to enable it. Any advice? > > > > > > > > Thank you! > > > > Konstantin > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj\_yKT6d8\_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj_yKT6d8_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj\_yKT6d8\_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj_yKT6d8_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [November 21, 2014, 3:41pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/6 "2014-11-21T15:41:39Z")
Disabling allocation helps, but it does not solve the problem completely. Just like Nik, one of my complaints (although not my primary one). ๐Ÿ™‚ I found that recovery gets easier when doing a rolling restart. First few servers always rebalance, the last ones do not. -- Ivan On Thu, Nov 20, 2014 at 9:51 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > You should disable allocation before you reboot, that will save a lot of > shard shuffling - > [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#rolling-upgrades) > > On 21 November 2014 13:48, Konstantin Erman [konste@gmail.com](mailto:konste@gmail.com) wrote: > > > I work on an experimental cluster of ES nodes running on Windows Server > > machines. Once in a while we have a need to reboot machines. The initial > > state - cluster is green and well balanced. One machine is gracefully taken > > offline and then after necessary service is performed it comes back online. > > All the hardware and file system content is intact. As soon as ES service > > starts on that machine, it assumes that there is no usable data locally and > > recovers as much data as it deems necessary for balancing from other nodes. > > > > This behavior puzzles me, because most of the data shards stored on that > > machine file system can be reused as they are. Cluster stores logs, so all > > indices except those for the current day never ever change until they get > > deleted. Can't ES node detect that it has perfect copies of some (actually > > most) of the shards and instead of copying them over just mark them as up > > to date? > > > > I suspect I don't know about some step to enable this behavior and I'm > > looking to enable it. Any advice? > > > > Thank you! > > Konstantin > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4fb2d8bc-7787-43e3-8c66-e241945d496b%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmc\_rMFzRUUrJSMJ9bY16tz-dZ8eSeUZobC7XaxWZTRPg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmc_rMFzRUUrJSMJ9bY16tz-dZ8eSeUZobC7XaxWZTRPg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmc\_rMFzRUUrJSMJ9bY16tz-dZ8eSeUZobC7XaxWZTRPg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmc_rMFzRUUrJSMJ9bY16tz-dZ8eSeUZobC7XaxWZTRPg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDjm%3DvBT7U3%3DQXwZzz83Bf52t21KYZwCuqdYgGJhXKuhQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDjm%3DvBT7U3%3DQXwZzz83Bf52t21KYZwCuqdYgGJhXKuhQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yves\_Dorfsman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yves_dorfsman/32/1106_2.png) [@Yves\_Dorfsman](https://discuss.elastic.co/u/Yves_Dorfsman)\ **Post date:** [November 21, 2014, 3:51pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/7 "2014-11-21T15:51:14Z")
Thanks Nicolas. Is this true on versions 0.9, or only on \> 1? I've had nodes die and restart, and they did copy everything! On 2014-11-20 22:02, Nikolas Everett wrote: > The thing is that this is a disk level operation. It pretty much rsyncs the > files from the current master shard to the node when it comes back online. > This would be OK if the replica shards matched the master but that is only > normally the case if the shard was moved to the node after it was mostly > complete and then you've had only a few writes. Normally shards don't match > each other because the way the index is maintained is nondeterministic. > > The translog replay is only used as a catch up after the rsync-like step. > > This is something that is being worked on. Its certainly my biggest complaint > about elasticsearch but I'm confident that it'll get better. > > Nik > > On Nov 20, 2014 11:11 PM, "Mark Walkom" \<[markwalkom@gmail.com](mailto:markwalkom@gmail.com) > [mailto:markwalkom@gmail.com](mailto:markwalkom@gmail.com)\> wrote: > > ``` > It will enter recovery where it syncs at the segment level from the > current primary, then the translog gets shipped over and (re)played, which > brings it all up to date. > > On 21 November 2014 14:51, Yves Dorfsman > wrote: > > If you do disable allocation before you reboot a node and a client > writes to a shard that had a replica on that node, does the entire > replica gets copied when the node come up? Or does it get just updated? > > On Thursday, 20 November 2014 19:52:26 UTC-7, Mark Walkom wrote: > > You should disable allocation before you reboot, that will save a > lot of shard shuffling - > http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/current/setup-__upgrade.html#rolling-upgrades > > > On 21 November 2014 13:48, Konstantin Erman wrote: > > I work on an experimental cluster of ES nodes running on > Windows Server machines. Once in a while we have a need to > reboot machines. The initial state - cluster is green and well > balanced. One machine is gracefully taken offline and then > after necessary service is performed it comes back online. All > the hardware and file system content is intact. As soon as ES > service starts on that machine, it assumes that there is no > usable data locally and recovers as much data as it deems > necessary for balancing from other nodes. > > This behavior puzzles me, because most of the data shards > stored on that machine file system can be reused as they are. > Cluster stores logs, so all indices except those for > the current day never ever change until they get deleted. > Can't ES node detect that it has perfect copies of some > (actually most) of the shards and instead of copying them over > just mark them as up to date? > > I suspect I don't know about some step to enable this behavior > and I'm looking to enable it. Any advice? > > Thank you! > Konstantin > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to elasticsearch+unsubscribe@googlegroups.com > . > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com > . > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscribe@googlegroups.com > . > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj_yKT6d8_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com > . > For more options, visit https://groups.google.com/d/optout. > > ``` > > -- > You received this message because you are subscribed to a topic in the Google > Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/ijQ1N-CAfc8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/ijQ1N-CAfc8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- > **[Yves Dorfsman](https://yves.zioup.com/)** > > Yves Dorfsman gpg: 4096R/32B0F416 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/546F5F72.3070002%40zioup.com](https://groups.google.com/d/msgid/elasticsearch/546F5F72.3070002%40zioup.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [November 21, 2014, 4:36pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/8 "2014-11-21T16:36:29Z")
Its been true for as long as I've used Elasticsearch which as .9X (can't remember which). Copying everything is pretty common because the index operations are non-deterministic so the files might not be the same at all. I pretty much assume I'm going to have to copy _almost_ everything every time I restart. It turns my rolling restarts into two day long affairs. I know its being worked on. Nik On Fri, Nov 21, 2014 at 10:51 AM, Yves Dorfsman [yves@zioup.com](mailto:yves@zioup.com) wrote: > Thanks Nicolas. > > Is this true on versions 0.9, or only on \> 1? > I've had nodes die and restart, and they did copy everything! > > On 2014-11-20 22:02, Nikolas Everett wrote: > > > The thing is that this is a disk level operation. It pretty much rsyncs > > the > > files from the current master shard to the node when it comes back > > online. > > This would be OK if the replica shards matched the master but that is > > only > > normally the case if the shard was moved to the node after it was mostly > > complete and then you've had only a few writes. Normally shards don't > > match > > each other because the way the index is maintained is nondeterministic. > > > > The translog replay is only used as a catch up after the rsync-like step. > > > > This is something that is being worked on. Its certainly my biggest > > complaint > > about elasticsearch but I'm confident that it'll get better. > > > > Nik > > > > On Nov 20, 2014 11:11 PM, "Mark Walkom" \<[markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > [mailto:markwalkom@gmail.com](mailto:markwalkom@gmail.com)\> wrote: > > > > ``` > > It will enter recovery where it syncs at the segment level from the > > current primary, then the translog gets shipped over and (re)played, > > > > ``` > > which > > > ``` > > brings it all up to date. > > > > On 21 November 2014 14:51, Yves Dorfsman > > wrote: > > > > If you do disable allocation before you reboot a node and a > > > > ``` > > client > > > ``` > > writes to a shard that had a replica on that node, does the > > > > ``` > > entire > > > ``` > > replica gets copied when the node come up? Or does it get just > > > > ``` > > updated? > > > ``` > > On Thursday, 20 November 2014 19:52:26 UTC-7, Mark Walkom wrote: > > > > You should disable allocation before you reboot, that will > > > > ``` > > save a > > > ``` > > lot of shard shuffling - > > > > ``` > > [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/current/setup-__upgrade.html#rolling-upgrades) > > > ``` > > < > > > > ``` > > [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#rolling-upgrades) > > > ``` > > On 21 November 2014 13:48, Konstantin Erman < > > > > ``` > > [kon...@gmail.com](mailto:kon...@gmail.com)\> wrote: > > > ``` > > I work on an experimental cluster of ES nodes running on > > Windows Server machines. Once in a while we have a need > > > > ``` > > to > > > ``` > > reboot machines. The initial state - cluster is green > > > > ``` > > and well > > > ``` > > balanced. One machine is gracefully taken offline and > > > > ``` > > then > > > ``` > > after necessary service is performed it comes back > > > > ``` > > online. All > > > ``` > > the hardware and file system content is intact. As soon > > > > ``` > > as ES > > > ``` > > service starts on that machine, it assumes that there is > > > > ``` > > no > > > ``` > > usable data locally and recovers as much data as it deems > > necessary for balancing from other nodes. > > > > This behavior puzzles me, because most of the data shards > > stored on that machine file system can be reused as they > > > > ``` > > are. > > > ``` > > Cluster stores logs, so all indices except those for > > the current day never ever change until they get deleted. > > Can't ES node detect that it has perfect copies of some > > (actually most) of the shards and instead of copying > > > > ``` > > them over > > > ``` > > just mark them as up to date? > > > > I suspect I don't know about some step to enable this > > > > ``` > > behavior > > > ``` > > and I'm looking to enable it. Any advice? > > > > Thank you! > > Konstantin > > > > -- > > You received this message because you are subscribed to the > > > > ``` > > Google > > > ``` > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from > > > > ``` > > it, send > > > ``` > > an email to elasticsearch+unsubscribe@googlegroups.com > > . > > To view this discussion on the web visit > > > > ``` > > [https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com) > > > ``` > > < > > > > ``` > > [https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > ``` > > -- > > You received this message because you are subscribed to the Google > > > > ``` > > Groups > > > ``` > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, > > > > ``` > > send an > > > ``` > > email to elasticsearch+unsubscribe@googlegroups.com > > . > > To view this discussion on the web visit > > > > ``` > > [https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj\_yKT6d8\_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj_yKT6d8_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com) > > > ``` > > < > > > > ``` > > [https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj\_yKT6d8\_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj_yKT6d8_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google > > Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/ijQ1N-CAfc8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/ijQ1N-CAfc8/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > > [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com) > > > \< > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > [http://yves.zioup.com](http://yves.zioup.com) > gpg: 4096R/32B0F416 > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/546F5F72.3070002%40zioup.com](https://groups.google.com/d/msgid/elasticsearch/546F5F72.3070002%40zioup.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0L1\_UGUxipKAL9R8BEUNfNGfdMxHXHOmMirZzDtp9xpg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0L1_UGUxipKAL9R8BEUNfNGfdMxHXHOmMirZzDtp9xpg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [November 22, 2014, 4:35pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/9 "2014-11-22T16:35:59Z")
Hi Konstantin, Check out [http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/](http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/) ## Otis Monitoring \* Alerting \* Anomaly Detection \* Centralized Log Management Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Thursday, November 20, 2014 9:48:56 PM UTC-5, Konstantin Erman wrote: > I work on an experimental cluster of ES nodes running on Windows Server > machines. Once in a while we have a need to reboot machines. The initial > state - cluster is green and well balanced. One machine is gracefully taken > offline and then after necessary service is performed it comes back online. > All the hardware and file system content is intact. As soon as ES service > starts on that machine, it assumes that there is no usable data locally and > recovers as much data as it deems necessary for balancing from other nodes. > > This behavior puzzles me, because most of the data shards stored on that > machine file system can be reused as they are. Cluster stores logs, so all > indices except those for the current day never ever change until they get > deleted. Can't ES node detect that it has perfect copies of some (actually > most) of the shards and instead of copying them over just mark them as up > to date? > > I suspect I don't know about some step to enable this behavior and I'm > looking to enable it. Any advice? > > Thank you! > Konstantin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6aeacdee-f8e9-405d-aca1-7166f1722fdd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6aeacdee-f8e9-405d-aca1-7166f1722fdd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Konstantin\_Erman](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@Konstantin\_Erman](https://discuss.elastic.co/u/Konstantin_Erman)\ **Post date:** [November 22, 2014, 6:43pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/10 "2014-11-22T18:43:45Z")
Yes, I have noticed that article right away, simply because I keep googling ES related questions every day ๐Ÿ™‚ Unfortunately the only practical advice I could learn from that article is to use doc\_values instead of field data and it does not really help with "full node rebuild after short down time" issue. One thing I noticed while watching Windows Resource Monitor during those lengthy node rebuilds though is that rebuilding node actively reads its shards, so there is a chance that it tries to bring its local data back to life and not just copies everything over from replicas, but unfortunately performance wise reviving local shards takes as long if not longer than dumb copying everything over from replicas. On Saturday, November 22, 2014 8:35:59 AM UTC-8, Otis Gospodnetic wrote: > Hi Konstantin, > > Check out [http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/](http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/) > > ## Otis > > Monitoring \* Alerting \* Anomaly Detection \* Centralized Log Management > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > On Thursday, November 20, 2014 9:48:56 PM UTC-5, Konstantin Erman wrote: > > > I work on an experimental cluster of ES nodes running on Windows Server > > machines. Once in a while we have a need to reboot machines. The initial > > state - cluster is green and well balanced. One machine is > > gracefully taken offline and then after necessary service is performed it > > comes back online. All the hardware and file system content is intact. As > > soon as ES service starts on that machine, it assumes that there is no > > usable data locally and recovers as much data as it deems necessary for > > balancing from other nodes. > > > > This behavior puzzles me, because most of the data shards stored on that > > machine file system can be reused as they are. Cluster stores logs, so all > > indices except those for the current day never ever change until they get > > deleted. Can't ES node detect that it has perfect copies of some (actually > > most) of the shards and instead of copying them over just mark them as up > > to date? > > > > I suspect I don't know about some step to enable this behavior and I'm > > looking to enable it. Any advice? > > > > Thank you! > > Konstantin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yves\_Dorfsman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yves_dorfsman/32/1106_2.png) [@Yves\_Dorfsman](https://discuss.elastic.co/u/Yves_Dorfsman)\ **Post date:** [November 22, 2014, 7:04pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/11 "2014-11-22T19:04:18Z")
On 2014-11-22 09:35, Otis Gospodnetic wrote: > Hi Konstantin, > > Check out [http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/](http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/) Good writing! Thanks. I wonder if there's any drawback from cutting indices in smaller (tiny?) shards? My thinking is this: We don't really change data in our bigger indices, we just keep adding to them, so ultimately as we re-build node, they should all have the same version of the old shards, which should make re-start, and even re-build from backups much faster. -- > **[Yves Dorfsman](https://yves.zioup.com/)** > > Yves Dorfsman gpg: 4096R/32B0F416 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5470DE32.5070902%40zioup.com](https://groups.google.com/d/msgid/elasticsearch/5470DE32.5070902%40zioup.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [November 22, 2014, 8:44pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/12 "2014-11-22T20:44:35Z")
Tiny shards have more ever head and aren't going to score results as accurately. On Nov 22, 2014 2:04 PM, "Yves Dorfsman" [yves@zioup.com](mailto:yves@zioup.com) wrote: > On 2014-11-22 09:35, Otis Gospodnetic wrote: > > > Hi Konstantin, > > > > Check out [http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/](http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/) > > Good writing! Thanks. > > I wonder if there's any drawback from cutting indices in smaller (tiny?) > shards? > > My thinking is this: We don't really change data in our bigger indices, we > just keep adding to them, so ultimately as we re-build node, they should > all > have the same version of the old shards, which should make re-start, and > even > re-build from backups much faster. > > -- > [http://yves.zioup.com](http://yves.zioup.com) > gpg: 4096R/32B0F416 > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/5470DE32.5070902%40zioup.com](https://groups.google.com/d/msgid/elasticsearch/5470DE32.5070902%40zioup.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2b6Omqo5K-fp%3DM4%3D-MJvV\_jfGjkCFHmu9u28g%3D5asJ6Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2b6Omqo5K-fp%3DM4%3D-MJvV_jfGjkCFHmu9u28g%3D5asJ6Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 23, 2014, 2:27pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/13 "2014-11-23T14:27:13Z")
FWIW with Lucene 5, segment and index checksums will arrive, and Elasticsearch will stop retrieving shards if local ones match remote ones. In the meantime you can increase indices.recovery.concurrent\_streams to something like 16 or even higher, so the recovery takes less than a minute. Jรถrg On Sat, Nov 22, 2014 at 7:43 PM, Konstantin Erman [konste@gmail.com](mailto:konste@gmail.com) wrote: > Yes, I have noticed that article right away, simply because I keep > googling ES related questions every day ๐Ÿ™‚ > > Unfortunately the only practical advice I could learn from that article is > to use doc\_values instead of field data and it does not really help with > "full node rebuild after short down time" issue. > > One thing I noticed while watching Windows Resource Monitor during those > lengthy node rebuilds though is that rebuilding node actively reads > its shards, so there is a chance that it tries to bring its local data back > to life and not just copies everything over from replicas, but > unfortunately performance wise reviving local shards takes as long if not > longer than dumb copying everything over from replicas. > > On Saturday, November 22, 2014 8:35:59 AM UTC-8, Otis Gospodnetic wrote: > > > Hi Konstantin, > > > > Check out [http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/](http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/) > > > > ## Otis > > > > Monitoring \* Alerting \* Anomaly Detection \* Centralized Log Management > > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > > > On Thursday, November 20, 2014 9:48:56 PM UTC-5, Konstantin Erman wrote: > > > > > I work on an experimental cluster of ES nodes running on Windows Server > > > machines. Once in a while we have a need to reboot machines. The initial > > > state - cluster is green and well balanced. One machine is > > > gracefully taken offline and then after necessary service is performed it > > > comes back online. All the hardware and file system content is intact. As > > > soon as ES service starts on that machine, it assumes that there is no > > > usable data locally and recovers as much data as it deems necessary for > > > balancing from other nodes. > > > > > > This behavior puzzles me, because most of the data shards stored on that > > > machine file system can be reused as they are. Cluster stores logs, so all > > > indices except those for the current day never ever change until they get > > > deleted. Can't ES node detect that it has perfect copies of some (actually > > > most) of the shards and instead of copying them over just mark them as up > > > to date? > > > > > > I suspect I don't know about some step to enable this behavior and I'm > > > looking to enable it. Any advice? > > > > > > Thank you! > > > Konstantin > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvQ3ZJz%3D7v\_QHTfgxsmznMTMLnsxGhrgiHuPBfTKoVWg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvQ3ZJz%3D7v_QHTfgxsmznMTMLnsxGhrgiHuPBfTKoVWg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Konstantin\_Erman](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@Konstantin\_Erman](https://discuss.elastic.co/u/Konstantin_Erman)\ **Post date:** [November 23, 2014, 8:01pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/14 "2014-11-23T20:01:17Z")
Advice to increase indices.recovery.concurrent\_streams sounds suspiciously specific to me ๐Ÿ™‚ What made you so confident that it is the bottleneck for recovery in most cases? And how cluster.routing.allocation.node\_concurrent\_recoveries should be set? On Sunday, November 23, 2014 6:27:40 AM UTC-8, Jรถrg Prante wrote: > FWIW with Lucene 5, segment and index checksums will arrive, and > Elasticsearch will stop retrieving shards if local ones match remote ones. > > In the meantime you can increase indices.recovery.concurrent\_streams to > something like 16 or even higher, so the recovery takes less than a minute. > > Jรถrg > > On Sat, Nov 22, 2014 at 7:43 PM, Konstantin Erman \<[kon...@gmail.com](mailto:kon...@gmail.com) > \\> wrote: > > > Yes, I have noticed that article right away, simply because I keep > > googling ES related questions every day ๐Ÿ™‚ > > > > Unfortunately the only practical advice I could learn from that article > > is to use doc\_values instead of field data and it does not really help with > > "full node rebuild after short down time" issue. > > > > One thing I noticed while watching Windows Resource Monitor during those > > lengthy node rebuilds though is that rebuilding node actively reads > > its shards, so there is a chance that it tries to bring its local data back > > to life and not just copies everything over from replicas, but > > unfortunately performance wise reviving local shards takes as long if not > > longer than dumb copying everything over from replicas. > > > > On Saturday, November 22, 2014 8:35:59 AM UTC-8, Otis Gospodnetic wrote: > > > > > Hi Konstantin, > > > > > > Check out [http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/](http://gibrown.com/2014/11/19/elasticsearch-the-broken-bits/) > > > > > > ## Otis > > > > > > Monitoring \* Alerting \* Anomaly Detection \* Centralized Log Management > > > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > > > > > On Thursday, November 20, 2014 9:48:56 PM UTC-5, Konstantin Erman wrote: > > > > > > > I work on an experimental cluster of ES nodes running on Windows Server > > > > machines. Once in a while we have a need to reboot machines. The initial > > > > state - cluster is green and well balanced. One machine is > > > > gracefully taken offline and then after necessary service is performed it > > > > comes back online. All the hardware and file system content is intact. As > > > > soon as ES service starts on that machine, it assumes that there is no > > > > usable data locally and recovers as much data as it deems necessary for > > > > balancing from other nodes. > > > > > > > > This behavior puzzles me, because most of the data shards stored on > > > > that machine file system can be reused as they are. Cluster stores logs, so > > > > all indices except those for the current day never ever change until they > > > > get deleted. Can't ES node detect that it has perfect copies of some > > > > (actually most) of the shards and instead of copying them over just mark > > > > them as up to date? > > > > > > > > I suspect I don't know about some step to enable this behavior and I'm > > > > looking to enable it. Any advice? > > > > > > > > Thank you! > > > > Konstantin > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/914332e3-2b14-49a4-be0e-5ddeadf9f357%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/462e3dc3-7e39-4cde-b00c-ad22377d40ad%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/462e3dc3-7e39-4cde-b00c-ad22377d40ad%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 23, 2014, 10:48pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/15 "2014-11-23T22:48:02Z")
The default indices recovery performance is limited by 3 concurrent streams and 20MB/sec. This is very slow on my machines. YMMV. Jรถrg On Sun, Nov 23, 2014 at 9:01 PM, Konstantin Erman [konste@gmail.com](mailto:konste@gmail.com) wrote: > Advice to increase indices.recovery.concurrent\_streams sounds > suspiciously specific to me ๐Ÿ™‚ What made you so confident that it is the > bottleneck for recovery in most cases? And how cluster.routing.allocation.node\_concurrent\_recoveries > should be set? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW\_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [November 24, 2014, 4:58pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/16 "2014-11-24T16:58:11Z")
It used to be 2 concurrent streams. Has the default been upped in recent versions? I agree, that number is awfully low. If you can disable indexing during rolling restarts, those numbers can be much higher. -- Ivan On Sun, Nov 23, 2014 at 5:48 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > The default indices recovery performance is limited by 3 concurrent > streams and 20MB/sec. This is very slow on my machines. YMMV. > > Jรถrg > > On Sun, Nov 23, 2014 at 9:01 PM, Konstantin Erman [konste@gmail.com](mailto:konste@gmail.com) > wrote: > > > Advice to increase indices.recovery.concurrent\_streams sounds > > suspiciously specific to me ๐Ÿ™‚ What made you so confident that it is the > > bottleneck for recovery in most cases? And how cluster.routing.allocation > > .node\_concurrent\_recoveries should be set? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW\_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW\_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAY-ucXNA6DeKpaVF9eMW%3Ddv95PsEZv8oPs0taojuLpxw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAY-ucXNA6DeKpaVF9eMW%3Ddv95PsEZv8oPs0taojuLpxw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Konstantin\_Erman](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@Konstantin\_Erman](https://discuss.elastic.co/u/Konstantin_Erman)\ **Post date:** [November 24, 2014, 6:53pm UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/17 "2014-11-24T18:53:28Z")
I don't believe we can disable indexing. ES cluster collects the logs from a huge server farm and those logs come in a steady rate. If we disable indexing we start dropping log events! Do I get it right? On Monday, November 24, 2014 8:58:21 AM UTC-8, Ivan Brusic wrote: > It used to be 2 concurrent streams. Has the default been upped in recent > versions? I agree, that number is awfully low. If you can disable indexing > during rolling restarts, those numbers can be much higher. > > -- > Ivan > > On Sun, Nov 23, 2014 at 5:48 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \ \< > [joerg...@gmail.com](mailto:joerg...@gmail.com) \\> wrote: > > > The default indices recovery performance is limited by 3 concurrent > > streams and 20MB/sec. This is very slow on my machines. YMMV. > > > > Jรถrg > > > > On Sun, Nov 23, 2014 at 9:01 PM, Konstantin Erman \<[kon...@gmail.com](mailto:kon...@gmail.com) > > \\> wrote: > > > > > Advice to increase indices.recovery.concurrent\_streams sounds > > > suspiciously specific to me ๐Ÿ™‚ What made you so confident that it is the > > > bottleneck for recovery in most cases? And how cluster.routing. > > > allocation.node\_concurrent\_recoveries should be set? > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW\_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW\_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH9BmW_nfZ3wGxOX0gNaJTDR2r0HSPeKc9dWNryUGP-Q%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fcbe921b-b381-4d92-93ed-81c6c32f0204%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fcbe921b-b381-4d92-93ed-81c6c32f0204%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:48am UTC](https://discuss.elastic.co/t/why-es-node-starts-recovering-all-the-data-from-other-nodes-after-reboot/20881/18 "2017-07-06T00:48:03Z")