# ES process restart causes full resync of all all shards to the restarted node **URL:** \ **Category:** Elasticsearch\ **Created:** [July 1, 2013, 1:13pm UTC](https://discuss.elastic.co/t/es-process-restart-causes-full-resync-of-all-all-shards-to-the-restarted-node/12600 "2013-07-01T13:13:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Piavlo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/piavlo/32/1214_2.png) [@Piavlo](https://discuss.elastic.co/u/Piavlo)\ **Post date:** [July 1, 2013, 1:13pm UTC](https://discuss.elastic.co/t/es-process-restart-causes-full-resync-of-all-all-shards-to-the-restarted-node/12600/1 "2013-07-01T13:13:41Z")
Hi, Each time I restart an ES process on any single node in the ES cluster, all shards to this node are rsynced from other nodes. This happens for all shards , even those that are not written to during the restart, why does it need to resync unmodified shards? This is causing very prolonged extra IO load and ES cluster indexing throughput suffers badly. Is there way to tell logstash not to resync unmodified shards? This is on 4 node cluster with replication factor of one, each index has 8 master shards and each node can have exactly 4 shards ``` "number_of_replicas" : 1, "number_of_shards" : 8, "index.routing.allocation.total_shards_per_node" : 4, ``` So no shard relocations happen then ES process restarted on one of the nodes. Thanks Alex -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![patrick\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick_2/32/2247_2.png) [@patrick\_2](https://discuss.elastic.co/u/patrick_2)\ **Post date:** [July 1, 2013, 3:27pm UTC](https://discuss.elastic.co/t/es-process-restart-causes-full-resync-of-all-all-shards-to-the-restarted-node/12600/2 "2013-07-01T15:27:28Z")
What we've started to do is disabling allocation during restarts: curl -XPUT localhost:9200/\_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable\_allocation": true}}' /etc/init.d/elasticsearch restart curl -XPUT localhost:9200/\_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable\_allocation": false}}' There is risk of data-loss should one of your servers go down but its way faster for recovery. HTH Patrick On Monday, July 1, 2013 3:13:41 PM UTC+2, Piavlo wrote: > Hi, > > Each time I restart an ES process on any single node in the ES cluster, > all shards to this node are rsynced from other nodes. > This happens for all shards , even those that are not written to during > the restart, why does it need to resync unmodified shards? > This is causing very prolonged extra IO load and ES cluster indexing > throughput suffers badly. > Is there way to tell logstash not to resync unmodified shards? > > This is on 4 node cluster with replication factor of one, each index has > 8 master shards and each node can have exactly 4 shards > > ``` > "number_of_replicas" : 1, > "number_of_shards" : 8, > "index.routing.allocation.total_shards_per_node" : 4, > > ``` > > So no shard relocations happen then ES process restarted on one of the > nodes. > > Thanks > Alex -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Piavlo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/piavlo/32/1214_2.png) [@Piavlo](https://discuss.elastic.co/u/Piavlo)\ **Post date:** [July 1, 2013, 10:31pm UTC](https://discuss.elastic.co/t/es-process-restart-causes-full-resync-of-all-all-shards-to-the-restarted-node/12600/3 "2013-07-01T22:31:34Z")
Hi Patrick, Afaiu this just prevents from resyncing shards to a restarted node, but then that node will just be up with zero shards? And once shard allocation is enabled again a resync will begin, no? Do you have an idea why ES tries to resync at all? as i'm preventing from shard reallocations during restart with "index.routing.allocation.total\_shards\_per\_node". And then node is back what makes ES think that it needs to resync even shards of indices which were never modified during a restart? Thanks Alex On Monday, July 1, 2013 6:27:28 PM UTC+3, [pat...@squirro.com](mailto:pat...@squirro.com) wrote: > What we've started to do is disabling allocation during restarts: > > curl -XPUT localhost:9200/\_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable\_allocation": > true}}' > /etc/init.d/elasticsearch restart > curl -XPUT localhost:9200/\_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable\_allocation": > false}}' > > There is risk of data-loss should one of your servers go down but its way > faster for recovery. > > HTH > > Patrick > > On Monday, July 1, 2013 3:13:41 PM UTC+2, Piavlo wrote: > > > Hi, > > > > Each time I restart an ES process on any single node in the ES cluster, > > all shards to this node are rsynced from other nodes. > > This happens for all shards , even those that are not written to during > > the restart, why does it need to resync unmodified shards? > > This is causing very prolonged extra IO load and ES cluster indexing > > throughput suffers badly. > > Is there way to tell logstash not to resync unmodified shards? > > > > This is on 4 node cluster with replication factor of one, each index has > > 8 master shards and each node can have exactly 4 shards > > > > ``` > > "number_of_replicas" : 1, > > "number_of_shards" : 8, > > "index.routing.allocation.total_shards_per_node" : 4, > > > > ``` > > > > So no shard relocations happen then ES process restarted on one of the > > nodes. > > > > Thanks > > Alex -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [July 2, 2013, 12:24am UTC](https://discuss.elastic.co/t/es-process-restart-causes-full-resync-of-all-all-shards-to-the-restarted-node/12600/4 "2013-07-02T00:24:20Z")
Hi Alex, Index shards are in sync at a high level, however, elasticsearch does not keep the low level lucene index files in sync as flushes/merges occur. You can verify this by restarting a node and once you are green again restart the same node. After the second restart all the lucene index files will already be in sync and it will be much faster to get to green. Then, the more data that is indexed, the longer it will take next time around as more segments get out of sync. Here's an old thread that discusses this: [https://groups.google.com/forum/#!msg/elasticsearch/XaVunruPg7M/BGeIdGX1XSUJ](https://groups.google.com/forum/#!msg/elasticsearch/XaVunruPg7M/BGeIdGX1XSUJ) I would love to see a feature to disable indexing, which guarantees nothing will get out of sync at a high level to allow a fast node recovery. Thanks! Paul On Monday, July 1, 2013 4:31:34 PM UTC-6, Piavlo wrote: > Hi Patrick, > > Afaiu this just prevents from resyncing shards to a restarted node, but > then that node will just be up with zero shards? > And once shard allocation is enabled again a resync will begin, no? > > Do you have an idea why ES tries to resync at all? as i'm preventing from > shard reallocations during restart > with "index.routing.allocation.total\_shards\_per\_node". And then node is > back what makes ES think that it needs to resync even shards > of indices which were never modified during a restart? > > Thanks > Alex > > On Monday, July 1, 2013 6:27:28 PM UTC+3, [pat...@squirro.com](mailto:pat...@squirro.com) wrote: > > > What we've started to do is disabling allocation during restarts: > > > > curl -XPUT localhost:9200/\_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable\_allocation": > > true}}' > > /etc/init.d/elasticsearch restart > > curl -XPUT localhost:9200/\_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable\_allocation": > > false}}' > > > > There is risk of data-loss should one of your servers go down but its way > > faster for recovery. > > > > HTH > > > > Patrick > > > > On Monday, July 1, 2013 3:13:41 PM UTC+2, Piavlo wrote: > > > > > Hi, > > > > > > Each time I restart an ES process on any single node in the ES cluster, > > > all shards to this node are rsynced from other nodes. > > > This happens for all shards , even those that are not written to during > > > the restart, why does it need to resync unmodified shards? > > > This is causing very prolonged extra IO load and ES cluster indexing > > > throughput suffers badly. > > > Is there way to tell logstash not to resync unmodified shards? > > > > > > This is on 4 node cluster with replication factor of one, each index has > > > 8 master shards and each node can have exactly 4 shards > > > > > > ``` > > > "number_of_replicas" : 1, > > > "number_of_shards" : 8, > > > "index.routing.allocation.total_shards_per_node" : 4, > > > > > > ``` > > > > > > So no shard relocations happen then ES process restarted on one of the > > > nodes. > > > > > > Thanks > > > Alex -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:28am UTC](https://discuss.elastic.co/t/es-process-restart-causes-full-resync-of-all-all-shards-to-the-restarted-node/12600/5 "2017-07-06T02:28:44Z")