Deleting Dangling nodes after upgrade

Elastic Stack Elasticsearch
1

When I upgraded from 0.17.6 to 0.19.0, after I restarted my cluster as 0.19
I get in ES log that there are dangling indices and they were marked for
removal after 2 minutes but all my nodes/indices had data before. ES is
claiming there is no metadata for them.

As I didn't set anything about gateway based on
thishttp://www.elasticsearch.org/guide/reference/modules/gateway/
[1]
I think i am using the
localhttp://www.elasticsearch.org/guide/reference/modules/gateway/local.html
[2]
gateway by default.

I know that when you start a empty cluster with a empty node, you needed to
set recover_after_nodes to assure a node with data comes before cluster
start

But as my cluster and nodes all already had data. What could be happening?

Do I need to start ES from the last Master node? Or something like?

Or am I accidentaly deleting the file for local gateway data in the update?
If this file exists where is it?

[1] http://www.elasticsearch.org/guide/reference/modules/gateway/ I think i
am using the local gateway by default
[2] http://www.elasticsearch.org/guide/reference/modules/gateway/local.html

2

The dangling indices message means that the index itself (its metadata) does not exists in the recovered global cluster metadata, but there seem to be evidence for them on disk. Can you try and restore the backup files created for the metadata, and start it back with 0.17.x, and then try and upgrade to 0.18 and then to 0.19. The upgrade path from 0.17 to 0.19 was not tested….

On Wednesday, March 14, 2012 at 12:48 AM, Onilton Maciel wrote:

When I upgraded from 0.17.6 to 0.19.0, after I restarted my cluster as 0.19
I get in ES log that there are dangling indices and they were marked for removal after 2 minutes but all my nodes/indices had data before. ES is claiming there is no metadata for them.

As I didn't set anything about gateway based on this (Elasticsearch Platform — Find real-time answers at scale | Elastic) [1] I think i am using the local (Elasticsearch Platform — Find real-time answers at scale | Elastic) [2] gateway by default.

I know that when you start a empty cluster with a empty node, you needed to set recover_after_nodes to assure a node with data comes before cluster start

But as my cluster and nodes all already had data. What could be happening?

Do I need to start ES from the last Master node? Or something like?

Or am I accidentaly deleting the file for local gateway data in the update? If this file exists where is it?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic I think i am using the local gateway by default
[2] Elasticsearch Platform — Find real-time answers at scale | Elastic

3

I don't know where is "backup files created for the metadata" you said.
Can you tell me where it is?

Also one thing I would like to know is:
If I upgrade from a direct version 0.17 -> 0.18 without using any gateway
(so it will be local by default), using the same nodes, same cluster, would
be any risk for me to lose my index or get dangling nodes?

PS: Assume I turned off my index program and flushed all the indexes before
turning off ES.

On Wed, Mar 14, 2012 at 8:22 AM, Shay Banon kimchy@gmail.com wrote:

The dangling indices message means that the index itself (its metadata)
does not exists in the recovered global cluster metadata, but there seem to
be evidence for them on disk. Can you try and restore the backup files
created for the metadata, and start it back with 0.17.x, and then try and
upgrade to 0.18 and then to 0.19. The upgrade path from 0.17 to 0.19 was
not tested….

On Wednesday, March 14, 2012 at 12:48 AM, Onilton Maciel wrote:

When I upgraded from 0.17.6 to 0.19.0, after I restarted my cluster as 0.19
I get in ES log that there are dangling indices and they were marked for
removal after 2 minutes but all my nodes/indices had data before. ES is
claiming there is no metadata for them.

As I didn't set anything about gateway based on thishttp://www.elasticsearch.org/guide/reference/modules/gateway/ [1]
I think i am using the localhttp://www.elasticsearch.org/guide/reference/modules/gateway/local.html [2]
gateway by default.

I know that when you start a empty cluster with a empty node, you needed
to set recover_after_nodes to assure a node with data comes before cluster
start

But as my cluster and nodes all already had data. What could be happening?

Do I need to start ES from the last Master node? Or something like?

Or am I accidentaly deleting the file for local gateway data in the
update? If this file exists where is it?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic I think
i am using the local gateway by default
[2]
Elasticsearch Platform — Find real-time answers at scale | Elastic

4

On 15 March 2012 01:37, Onilton Maciel oniltonmaciel@gmail.com wrote:

I don't know where is "backup files created for the metadata" you said.
Can you tell me where it is?

Also one thing I would like to know is:
If I upgrade from a direct version 0.17 -> 0.18 without using any gateway
(so it will be local by default), using the same nodes, same cluster, would
be any risk for me to lose my index or get dangling nodes?

PS: Assume I turned off my index program and flushed all the indexes
before turning off ES.

On Wed, Mar 14, 2012 at 8:22 AM, Shay Banon kimchy@gmail.com wrote:

The dangling indices message means that the index itself (its metadata)
does not exists in the recovered global cluster metadata, but there seem to
be evidence for them on disk. Can you try and restore the backup files
created for the

I've been meaning to post more details about this myself, but we've also
been running 0.17.9 in prod for a while, and planned to jump to 0.19, but
found we must upgrade to 0.18.x, and then to 0.19 for the existing
indexes to restore correctly.