Re-sizing a cluster - Shards and Replicas - Purge?


(Tony Su) #1

The following has bearing on understanding what happens to the data store
when Replicas are increased or decreased.

Reading about re-sizing the cluster, I've read

  • Changing the number of shards requires re-indexing, so it's highly
    recommended to allocate a number of shards consistent with the max number
    of expected nodes in the cluster from the beginning.

  • Replicas can be added or removed at any time without an unusual load.

So, it begs the question, I would expect that adding replicas creates new
copies.
But, what happens if the number of replicas is decreased?
Is there a difference changing replica number by runtime command compared
to starting up a node (elasticsearch.yml), particularly if replica numbers
are decreased?
I assume that the replica number with the highest number is removed first?
And what about the data, is it automatically purged or left "orphaned" in
data storage?

And if orphaned in data storage, is there a recommended way to manually
remove the replica data no longer used?
I would also expect that after decreasing the number of replicas, if the
number is increased again that whole new replicas would be created and
allocated, none of the old replicas might be checked for integrity and
possibly re-used (same procedure as recovery).

Thx,
Tony

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bcca6dc6-e3b7-4cfb-ae70-53ecae82e4ea%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Binh Ly) #2

When you decrease the replicas dynamically, the data is deleted from
wherever nodes they reside in. Then when you increase it back dynamically,
ES will then make new copies of the shards on-the-fly. Also when you set
index.number_of_replicas in the YML file, that only applies to new indexes
created after start up. If you already had an existing index prior to that
change in the YML file, no change will be made to the replica count of that
prior existing index.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8ed8fb2f-5c75-48c3-b90d-4ae3dcc99e88%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Tony Su) #3

Thank you for what appears to be a complete explanation describing
all scenarios, both runtime changes and startup (based on
elasticsearch.yml).
Enlightening that there are significant differences.

Tony

On Tuesday, February 18, 2014 7:48:16 AM UTC-8, Binh Ly wrote:

When you decrease the replicas dynamically, the data is deleted from
wherever nodes they reside in. Then when you increase it back dynamically,
ES will then make new copies of the shards on-the-fly. Also when you set
index.number_of_replicas in the YML file, that only applies to new indexes
created after start up. If you already had an existing index prior to that
change in the YML file, no change will be made to the replica count of that
prior existing index.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b9f8ce44-0de3-4c6e-968c-40d192cd42d9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #4