Dangling index, exists on local file system, but not in cluster metadata, scheduling to delete

Elastic Stack Elasticsearch
1

Hi,
My index is missing. How can I find it back?
Firstly, I created an index on one node(2 shards, no replica). Then
shutdown the node, and add another node into the cluster. And change the
configure to 2 shards+ 1 replica.
After the both nodes comes up, the previous index is missing.
From the log, it reported: dangling index, exists on local file system,
but not in cluster metadata, scheduling to delete
How can I recover my index?
Thanks.

2

You mean that you :

  1. start a node (node1)
  2. create an index
  3. stop the node
  4. start a new node (node2)
  5. send a request to "modify" the index

If you have done this, your index in node1 does not exist anymore.

Because you start a new Node (node2) without having node1 running. So Node1 can
not talk with node2 to replicate its metadata and data.

So, Node2 is considering itself as a master of your cluster.
When you ask to "modify" index, in fact, Node2 doesn't know that this index
should already exists and create a new index with 2 shards and 1 replica.

When Node1 wake up again, it gets an information from the master (Node2) that
the real version of your index is the one Node2 knows...

So, don't add a new node to your cluster when all the other nodes of this
cluster are down.

My 2 cents
David.

Le 5 juin 2012 à 08:46, jackiedong jackiedong168@gmail.com a écrit :

Hi,
My index is missing. How can I find it back?
Firstly, I created an index on one node(2 shards, no replica). Then
shutdown the node, and add another node into the cluster. And change the
configure to 2 shards+ 1 replica.
After the both nodes comes up, the previous index is missing.
From the log, it reported: dangling index, exists on local file system,
but not in cluster metadata, scheduling to delete
How can I recover my index?
Thanks.

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

3

I mean:

  1. start a node1
  2. create an index and insert some datas
  3. stop the node1
  4. modify the configure to let node1 and node2 in the same cluster and both
    set replica to 1
  5. start the node1 and node2 at the same time
  6. after the node1 and node2 starts, check the ES head, no index data is
    shown.
    But I checked the node1's data directory. My index data is still there.

How can I bring back my index data? Or will my index lose totally?
Thanks.

-Regards-
-Jackie-

在 2012年6月5日星期二UTC+8下午2时59分15秒，David Pilato写道：

You mean that you :

  1. start a node (node1)

  2. create an index

  3. stop the node

  4. start a new node (node2)

  5. send a request to "modify" the index

If you have done this, your index in node1 does not exist anymore.

Because you start a new Node (node2) without having node1 running. So
Node1 can not talk with node2 to replicate its metadata and data.

So, Node2 is considering itself as a master of your cluster.

When you ask to "modify" index, in fact, Node2 doesn't know that this
index should already exists and create a new index with 2 shards and 1
replica.

When Node1 wake up again, it gets an information from the master (Node2)
that the real version of your index is the one Node2 knows...

So, don't add a new node to your cluster when all the other nodes of this
cluster are down.

My 2 cents

David.

Le 5 juin 2012 à 08:46, jackiedong jackiedong168@gmail.com a écrit :

Hi,
My index is missing. How can I find it back?
Firstly, I created an index on one node(2 shards, no replica). Then
shutdown the node, and add another node into the cluster. And change the
configure to 2 shards+ 1 replica.
After the both nodes comes up, the previous index is missing.
From the log, it reported: dangling index, exists on local file
system, but not in cluster metadata, scheduling to delete
How can I recover my index?
Thanks.

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

4

I still don't understand the process..., you create an index with no
replicas on 2 nodes, then stop node1, and then bring it back?

On Tue, Jun 5, 2012 at 10:32 AM, jackiedong jackiedong168@gmail.com wrote:

I mean:

  1. start a node1
  2. create an index and insert some datas
  3. stop the node1
  4. modify the configure to let node1 and node2 in the same cluster and
    both set replica to 1
  5. start the node1 and node2 at the same time
  6. after the node1 and node2 starts, check the ES head, no index data is
    shown.
    But I checked the node1's data directory. My index data is still there.

How can I bring back my index data? Or will my index lose totally?
Thanks.

-Regards-
-Jackie-

在 2012年6月5日星期二UTC+8下午2时59分15秒，David Pilato写道：

You mean that you :

  1. start a node (node1)

  2. create an index

  3. stop the node

  4. start a new node (node2)

  5. send a request to "modify" the index

If you have done this, your index in node1 does not exist anymore.

Because you start a new Node (node2) without having node1 running. So
Node1 can not talk with node2 to replicate its metadata and data.

So, Node2 is considering itself as a master of your cluster.

When you ask to "modify" index, in fact, Node2 doesn't know that this
index should already exists and create a new index with 2 shards and 1
replica.

When Node1 wake up again, it gets an information from the master (Node2)
that the real version of your index is the one Node2 knows...

So, don't add a new node to your cluster when all the other nodes of this
cluster are down.

My 2 cents

David.

Le 5 juin 2012 à 08:46, jackiedong jackiedong168@gmail.com a écrit :

Hi,
My index is missing. How can I find it back?
Firstly, I created an index on one node(2 shards, no replica). Then
shutdown the node, and add another node into the cluster. And change the
configure to 2 shards+ 1 replica.
After the both nodes comes up, the previous index is missing.
From the log, it reported: dangling index, exists on local file
system, but not in cluster metadata, scheduling to delete
How can I recover my index?
Thanks.

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

5

I think what he did is,

  • on node1 he created an index "X".
  • stopped node1
  • configured node1 and node2 to be in the same cluster and index.number_of_replicas of each index to 1.
  • start node1 and node2 at the same time.

I think when he indexed "X", he didn't set up index.number_of_replicas yet. So, the index will only reside in node1 after starting both nodes at the same time. If he apply the update settings on index "X" using API after starting the nodes, "X" will get replicated to the other node.

Cheers
Lollypop