ES process restart causes full resync of all all shards to the restarted node

Hi,

Each time I restart an ES process on any single node in the ES cluster,
all shards to this node are rsynced from other nodes.
This happens for all shards , even those that are not written to during
the restart, why does it need to resync unmodified shards?
This is causing very prolonged extra IO load and ES cluster indexing
throughput suffers badly.
Is there way to tell logstash not to resync unmodified shards?

This is on 4 node cluster with replication factor of one, each index has
8 master shards and each node can have exactly 4 shards

     "number_of_replicas" : 1,
     "number_of_shards" : 8,
     "index.routing.allocation.total_shards_per_node" : 4,

So no shard relocations happen then ES process restarted on one of the
nodes.

Thanks
Alex

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

What we've started to do is disabling allocation during restarts:

curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable_allocation":
true}}'
/etc/init.d/elasticsearch restart
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable_allocation":
false}}'

There is risk of data-loss should one of your servers go down but its way
faster for recovery.

HTH

Patrick

On Monday, July 1, 2013 3:13:41 PM UTC+2, Piavlo wrote:

Hi,

Each time I restart an ES process on any single node in the ES cluster,
all shards to this node are rsynced from other nodes.
This happens for all shards , even those that are not written to during
the restart, why does it need to resync unmodified shards?
This is causing very prolonged extra IO load and ES cluster indexing
throughput suffers badly.
Is there way to tell logstash not to resync unmodified shards?

This is on 4 node cluster with replication factor of one, each index has
8 master shards and each node can have exactly 4 shards

     "number_of_replicas" : 1, 
     "number_of_shards" : 8, 
     "index.routing.allocation.total_shards_per_node" : 4, 

So no shard relocations happen then ES process restarted on one of the
nodes.

Thanks
Alex

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Patrick,

Afaiu this just prevents from resyncing shards to a restarted node, but
then that node will just be up with zero shards?
And once shard allocation is enabled again a resync will begin, no?

Do you have an idea why ES tries to resync at all? as i'm preventing from
shard reallocations during restart
with "index.routing.allocation.total_shards_per_node". And then node is
back what makes ES think that it needs to resync even shards
of indices which were never modified during a restart?

Thanks
Alex

On Monday, July 1, 2013 6:27:28 PM UTC+3, pat...@squirro.com wrote:

What we've started to do is disabling allocation during restarts:

curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable_allocation":
true}}'
/etc/init.d/elasticsearch restart
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable_allocation":
false}}'

There is risk of data-loss should one of your servers go down but its way
faster for recovery.

HTH

Patrick

On Monday, July 1, 2013 3:13:41 PM UTC+2, Piavlo wrote:

Hi,

Each time I restart an ES process on any single node in the ES cluster,
all shards to this node are rsynced from other nodes.
This happens for all shards , even those that are not written to during
the restart, why does it need to resync unmodified shards?
This is causing very prolonged extra IO load and ES cluster indexing
throughput suffers badly.
Is there way to tell logstash not to resync unmodified shards?

This is on 4 node cluster with replication factor of one, each index has
8 master shards and each node can have exactly 4 shards

     "number_of_replicas" : 1, 
     "number_of_shards" : 8, 
     "index.routing.allocation.total_shards_per_node" : 4, 

So no shard relocations happen then ES process restarted on one of the
nodes.

Thanks
Alex

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Alex,
Index shards are in sync at a high level, however, elasticsearch does not
keep the low level lucene index files in sync as flushes/merges occur.

You can verify this by restarting a node and once you are green again
restart the same node. After the second restart all the lucene index files
will already be in sync and it will be much faster to get to green. Then,
the more data that is indexed, the longer it will take next time around as
more segments get out of sync.

Here's an old thread that discusses this:
https://groups.google.com/forum/#!msg/elasticsearch/XaVunruPg7M/BGeIdGX1XSUJ

I would love to see a feature to disable indexing, which guarantees nothing
will get out of sync at a high level to allow a fast node recovery.

Thanks!
Paul

On Monday, July 1, 2013 4:31:34 PM UTC-6, Piavlo wrote:

Hi Patrick,

Afaiu this just prevents from resyncing shards to a restarted node, but
then that node will just be up with zero shards?
And once shard allocation is enabled again a resync will begin, no?

Do you have an idea why ES tries to resync at all? as i'm preventing from
shard reallocations during restart
with "index.routing.allocation.total_shards_per_node". And then node is
back what makes ES think that it needs to resync even shards
of indices which were never modified during a restart?

Thanks
Alex

On Monday, July 1, 2013 6:27:28 PM UTC+3, pat...@squirro.com wrote:

What we've started to do is disabling allocation during restarts:

curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable_allocation":
true}}'
/etc/init.d/elasticsearch restart
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable_allocation":
false}}'

There is risk of data-loss should one of your servers go down but its way
faster for recovery.

HTH

Patrick

On Monday, July 1, 2013 3:13:41 PM UTC+2, Piavlo wrote:

Hi,

Each time I restart an ES process on any single node in the ES cluster,
all shards to this node are rsynced from other nodes.
This happens for all shards , even those that are not written to during
the restart, why does it need to resync unmodified shards?
This is causing very prolonged extra IO load and ES cluster indexing
throughput suffers badly.
Is there way to tell logstash not to resync unmodified shards?

This is on 4 node cluster with replication factor of one, each index has
8 master shards and each node can have exactly 4 shards

     "number_of_replicas" : 1, 
     "number_of_shards" : 8, 
     "index.routing.allocation.total_shards_per_node" : 4, 

So no shard relocations happen then ES process restarted on one of the
nodes.

Thanks
Alex

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.