Old shards on re-joining nodes useful?


(Yongtao You) #1

Hi,

Quick question, please. If a node re-joins the cluster after it got
rebooted a few minutes ago, will ES make use of the old shards on this node
under any circumstances? Are they ever going to be used? For example, if a
shard does not exist anywhere else other than an old copy on this node,
will it be used?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/88ac2fc3-ff72-4895-86cf-bbeb66937b3f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Old shards not deleted upon relocation
(Jörg Prante) #2

Yes, if the recovery of an index succeeds, the shards of the rejoined node
for the index will be used. Do you mean orphaned shards, where the index
does no longer exist?

Jörg

On Mon, Jun 23, 2014 at 7:26 PM, Yongtao You yongtao.you@gmail.com wrote:

Hi,

Quick question, please. If a node re-joins the cluster after it got
rebooted a few minutes ago, will ES make use of the old shards on this node
under any circumstances? Are they ever going to be used? For example, if a
shard does not exist anywhere else other than an old copy on this node,
will it be used?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/88ac2fc3-ff72-4895-86cf-bbeb66937b3f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/88ac2fc3-ff72-4895-86cf-bbeb66937b3f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFDj9%2BSSP8T7wirOJA_hLyp5eNJvtc23VkXSTL-P3592Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Yongtao You) #3

So upon rebooting of a node, I should NOT attempt to remove all the
existing ES data before re-joining the cluster because these data might be
useful?

Yes, I meant orphaned shards.

Thanks.
Yongtao

On Monday, June 23, 2014 12:02:53 PM UTC-7, Jörg Prante wrote:

Yes, if the recovery of an index succeeds, the shards of the rejoined node
for the index will be used. Do you mean orphaned shards, where the index
does no longer exist?

Jörg

On Mon, Jun 23, 2014 at 7:26 PM, Yongtao You <yongt...@gmail.com
<javascript:>> wrote:

Hi,

Quick question, please. If a node re-joins the cluster after it got
rebooted a few minutes ago, will ES make use of the old shards on this node
under any circumstances? Are they ever going to be used? For example, if a
shard does not exist anywhere else other than an old copy on this node,
will it be used?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/88ac2fc3-ff72-4895-86cf-bbeb66937b3f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/88ac2fc3-ff72-4895-86cf-bbeb66937b3f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/292f400c-90eb-4e51-9348-7152832d5331%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #4

No, you must not remove any data. There are several options what ES can do
with orphaned shards:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway-local.html

Example of a log entry when orphaned shard is detected:

[2014-06-23 21:46:05,841][INFO ][gateway.local.state.meta ] [Lonnie
Thompson Lincoln] [test] dangling index, exists on local file system, but
not in cluster metadata, scheduling to delete in [2h], auto import to
cluster state [YES]

You can test this very easily by starting two nodes, index two docs, stop
one node, delete the index, and start the stopped node again.

Jörg

On Mon, Jun 23, 2014 at 9:18 PM, Yongtao You yongtao.you@gmail.com wrote:

So upon rebooting of a node, I should NOT attempt to remove all the
existing ES data before re-joining the cluster because these data might be
useful?

Yes, I meant orphaned shards.

Thanks.
Yongtao

On Monday, June 23, 2014 12:02:53 PM UTC-7, Jörg Prante wrote:

Yes, if the recovery of an index succeeds, the shards of the rejoined
node for the index will be used. Do you mean orphaned shards, where the
index does no longer exist?

Jörg

On Mon, Jun 23, 2014 at 7:26 PM, Yongtao You yongt...@gmail.com wrote:

Hi,

Quick question, please. If a node re-joins the cluster after it got
rebooted a few minutes ago, will ES make use of the old shards on this node
under any circumstances? Are they ever going to be used? For example, if a
shard does not exist anywhere else other than an old copy on this node,
will it be used?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/88ac2fc3-ff72-4895-86cf-bbeb66937b3f%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/88ac2fc3-ff72-4895-86cf-bbeb66937b3f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/292f400c-90eb-4e51-9348-7152832d5331%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/292f400c-90eb-4e51-9348-7152832d5331%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH3fL9K3C3w4ajLcXEymatXwFpWOGhB79aMMuj5OCPi%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Yongtao You) #5

Thank you, Jörg.

On Monday, June 23, 2014 12:59:12 PM UTC-7, Jörg Prante wrote:

No, you must not remove any data. There are several options what ES can do
with orphaned shards:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway-local.html

Example of a log entry when orphaned shard is detected:

[2014-06-23 21:46:05,841][INFO ][gateway.local.state.meta ] [Lonnie
Thompson Lincoln] [test] dangling index, exists on local file system, but
not in cluster metadata, scheduling to delete in [2h], auto import to
cluster state [YES]

You can test this very easily by starting two nodes, index two docs, stop
one node, delete the index, and start the stopped node again.

Jörg

On Mon, Jun 23, 2014 at 9:18 PM, Yongtao You <yongt...@gmail.com
<javascript:>> wrote:

So upon rebooting of a node, I should NOT attempt to remove all the
existing ES data before re-joining the cluster because these data might be
useful?

Yes, I meant orphaned shards.

Thanks.
Yongtao

On Monday, June 23, 2014 12:02:53 PM UTC-7, Jörg Prante wrote:

Yes, if the recovery of an index succeeds, the shards of the rejoined
node for the index will be used. Do you mean orphaned shards, where the
index does no longer exist?

Jörg

On Mon, Jun 23, 2014 at 7:26 PM, Yongtao You yongt...@gmail.com wrote:

Hi,

Quick question, please. If a node re-joins the cluster after it got
rebooted a few minutes ago, will ES make use of the old shards on this node
under any circumstances? Are they ever going to be used? For example, if a
shard does not exist anywhere else other than an old copy on this node,
will it be used?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/88ac2fc3-ff72-4895-86cf-bbeb66937b3f%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/88ac2fc3-ff72-4895-86cf-bbeb66937b3f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/292f400c-90eb-4e51-9348-7152832d5331%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/292f400c-90eb-4e51-9348-7152832d5331%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/16091ec8-b53a-492d-972c-d61ab381aa4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #6