Dangling index info messages in logs

Elastic Stack Elasticsearch
1

Hello all,

I have an elasticsearch cluster of 16 nodes , all of which are eligible
master and data nodes. I had to add some load balancing nodes to the
cluster so i added 6 extra nodes to the cluster . These newly added nodes
are all eligible master nodes only(not eligible data nodes). I restarted
the whole cluster by making the original 16 nodes to be eligible data nodes
only and the newly added 6 nodes were kept as eligible master nodes only.
During the first restart after changing the whole configuration of cluster
I started getting these INFO messages in the logs :
[2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user%
index] dangling index, exists on local file system, but not in cluster
metadata, scheduling to delete in [2h], auto import to cluster state [YES]

I thought the indices were about to be deleted so I immediately shutdown
the cluster and reverted back to the original configuration.
Can anyone shed a light on these info messages? According to my
understanding, only the cluster meta data of theses indices were about to
changed or deleted but not the data in the index. It would be great if
anyone could give a quick explanation about why I am getting these messages.
Thanks in advance!!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fa169794-6d3c-48a9-b98c-69d4c60aae58%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Oh I forgot to mention... I am using Elasticsearch version 1.0.1
Thanks!

On Saturday, March 1, 2014 9:42:04 PM UTC-8, hrishikesh prabhune wrote:

Hello all,

I have an elasticsearch cluster of 16 nodes , all of which are eligible
master and data nodes. I had to add some load balancing nodes to the
cluster so i added 6 extra nodes to the cluster . These newly added nodes
are all eligible master nodes only(not eligible data nodes). I restarted
the whole cluster by making the original 16 nodes to be eligible data nodes
only and the newly added 6 nodes were kept as eligible master nodes only.
During the first restart after changing the whole configuration of cluster
I started getting these INFO messages in the logs :
[2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user
%index] dangling index, exists on local file system, but not in cluster
metadata, scheduling to delete in [2h], auto import to cluster state [YES]

I thought the indices were about to be deleted so I immediately shutdown
the cluster and reverted back to the original configuration.
Can anyone shed a light on these info messages? According to my
understanding, only the cluster meta data of theses indices were about to
changed or deleted but not the data in the index. It would be great if
anyone could give a quick explanation about why I am getting these messages.
Thanks in advance!!

On Saturday, March 1, 2014 9:42:04 PM UTC-8, hrishikesh prabhune wrote:

Hello all,

I have an elasticsearch cluster of 16 nodes , all of which are eligible
master and data nodes. I had to add some load balancing nodes to the
cluster so i added 6 extra nodes to the cluster . These newly added nodes
are all eligible master nodes only(not eligible data nodes). I restarted
the whole cluster by making the original 16 nodes to be eligible data nodes
only and the newly added 6 nodes were kept as eligible master nodes only.
During the first restart after changing the whole configuration of cluster
I started getting these INFO messages in the logs :
[2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user
%index] dangling index, exists on local file system, but not in cluster
metadata, scheduling to delete in [2h], auto import to cluster state [YES]

I thought the indices were about to be deleted so I immediately shutdown
the cluster and reverted back to the original configuration.
Can anyone shed a light on these info messages? According to my
understanding, only the cluster meta data of theses indices were about to
changed or deleted but not the data in the index. It would be great if
anyone could give a quick explanation about why I am getting these messages.
Thanks in advance!!

On Saturday, March 1, 2014 9:42:04 PM UTC-8, hrishikesh prabhune wrote:

Hello all,

I have an elasticsearch cluster of 16 nodes , all of which are eligible
master and data nodes. I had to add some load balancing nodes to the
cluster so i added 6 extra nodes to the cluster . These newly added nodes
are all eligible master nodes only(not eligible data nodes). I restarted
the whole cluster by making the original 16 nodes to be eligible data nodes
only and the newly added 6 nodes were kept as eligible master nodes only.
During the first restart after changing the whole configuration of cluster
I started getting these INFO messages in the logs :
[2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user
%index] dangling index, exists on local file system, but not in cluster
metadata, scheduling to delete in [2h], auto import to cluster state [YES]

I thought the indices were about to be deleted so I immediately shutdown
the cluster and reverted back to the original configuration.
Can anyone shed a light on these info messages? According to my
understanding, only the cluster meta data of theses indices were about to
changed or deleted but not the data in the index. It would be great if
anyone could give a quick explanation about why I am getting these messages.
Thanks in advance!!

On Saturday, March 1, 2014 9:42:04 PM UTC-8, hrishikesh prabhune wrote:

Hello all,

I have an elasticsearch cluster of 16 nodes , all of which are eligible
master and data nodes. I had to add some load balancing nodes to the
cluster so i added 6 extra nodes to the cluster . These newly added nodes
are all eligible master nodes only(not eligible data nodes). I restarted
the whole cluster by making the original 16 nodes to be eligible data nodes
only and the newly added 6 nodes were kept as eligible master nodes only.
During the first restart after changing the whole configuration of cluster
I started getting these INFO messages in the logs :
[2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user
%index] dangling index, exists on local file system, but not in cluster
metadata, scheduling to delete in [2h], auto import to cluster state [YES]

I thought the indices were about to be deleted so I immediately shutdown
the cluster and reverted back to the original configuration.
Can anyone shed a light on these info messages? According to my
understanding, only the cluster meta data of theses indices were about to
changed or deleted but not the data in the index. It would be great if
anyone could give a quick explanation about why I am getting these messages.
Thanks in advance!!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/74ace4dd-b77d-48ea-8622-033f903e2e51%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

The problem here is that the new master was not synchronized with an old master.
So he started with an empty cluster state.

To do that kind of operation, you have to bring your new master nodes up in a running cluster so they will be synchronized, then stop the cluster, change data nodes settings and restart.

You should test on a dev machine what would happen if you set again old node to be master again. I think it won't work and will still try to remove your old data.
If you have a backup somewhere of your cluster state, that would probably be useful.

My 2 cents.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 mars 2014 à 06:42, hrishikesh prabhune hruship@gmail.com a écrit :

Hello all,

I have an elasticsearch cluster of 16 nodes , all of which are eligible master and data nodes. I had to add some load balancing nodes to the cluster so i added 6 extra nodes to the cluster . These newly added nodes are all eligible master nodes only(not eligible data nodes). I restarted the whole cluster by making the original 16 nodes to be eligible data nodes only and the newly added 6 nodes were kept as eligible master nodes only.
During the first restart after changing the whole configuration of cluster I started getting these INFO messages in the logs :
[2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user%index] dangling index, exists on local file system, but not in cluster metadata, scheduling to delete in [2h], auto import to cluster state [YES]

I thought the indices were about to be deleted so I immediately shutdown the cluster and reverted back to the original configuration.
Can anyone shed a light on these info messages? According to my understanding, only the cluster meta data of theses indices were about to changed or deleted but not the data in the index. It would be great if anyone could give a quick explanation about why I am getting these messages.
Thanks in advance!!

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fa169794-6d3c-48a9-b98c-69d4c60aae58%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48B542DA-6856-4D57-8693-FC680225BF17%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

4

Thanks for a quick reply!
Thankfully I didn't see any data loss. I think its because when I saw the
dangling index error I shutdown the whole cluster immediately before the
cluster would start recovering the indices. And then I reverted back to the
original configuration of the cluster. After that everything recovered
smoothly.
Also the INFO message stated that the delete was scheduled in 2 hours. But
no such delete happened when I reverted back to the old cluster
configuration. Does that mean that the old cluster meta information did not
get overwritten? I am still kind of confused why didn't the auto import to
cluster state didnt erase the old cluster state. Can you please elaborate a
little bit more on the this particular line in the INFO message

scheduling to delete in [2h], auto import to cluster state [YES]

On Sat, Mar 1, 2014 at 11:57 PM, David Pilato david@pilato.fr wrote:

The problem here is that the new master was not synchronized with an old
master.
So he started with an empty cluster state.

To do that kind of operation, you have to bring your new master nodes up
in a running cluster so they will be synchronized, then stop the cluster,
change data nodes settings and restart.

You should test on a dev machine what would happen if you set again old
node to be master again. I think it won't work and will still try to remove
your old data.
If you have a backup somewhere of your cluster state, that would probably
be useful.

My 2 cents.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 mars 2014 à 06:42, hrishikesh prabhune hruship@gmail.com a écrit :

Hello all,

I have an elasticsearch cluster of 16 nodes , all of which are eligible
master and data nodes. I had to add some load balancing nodes to the
cluster so i added 6 extra nodes to the cluster . These newly added nodes
are all eligible master nodes only(not eligible data nodes). I restarted
the whole cluster by making the original 16 nodes to be eligible data nodes
only and the newly added 6 nodes were kept as eligible master nodes only.
During the first restart after changing the whole configuration of cluster
I started getting these INFO messages in the logs :
[2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user
%index] dangling index, exists on local file system, but not in cluster
metadata, scheduling to delete in [2h], auto import to cluster state [YES]

I thought the indices were about to be deleted so I immediately shutdown
the cluster and reverted back to the original configuration.
Can anyone shed a light on these info messages? According to my
understanding, only the cluster meta data of theses indices were about to
changed or deleted but not the data in the index. It would be great if
anyone could give a quick explanation about why I am getting these messages.
Thanks in advance!!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fa169794-6d3c-48a9-b98c-69d4c60aae58%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/WXAoavhplNY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/48B542DA-6856-4D57-8693-FC680225BF17%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BSd%2B_fpxeRnskF6FDb6hTK_FaaJj4zZEDLnip17bNO1LajD%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.