Deleted index shows up after cluster restart

Hi,
We are maintaining some daily indexes in multi-nodes cluster. The old
indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?

Thanks!
Jaguar

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Some more detail, this is with version 0.90. Not sure if it's the same for
previous version.

在 2013年7月16日星期二UTC+8上午11时39分58秒,Jaguar写道:

Hi,
We are maintaining some daily indexes in multi-nodes cluster. The old
indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?

Thanks!
Jaguar

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You potentially had a split brain at one point and indexed data to two
different clusters. Or you had two instances running on the same server.
You should double check that there is only one directory called 0 under the
data directory.

--
Ivan

On Mon, Jul 15, 2013 at 8:40 PM, Jaguar xiong.jaguar@gmail.com wrote:

Some more detail, this is with version 0.90. Not sure if it's the same for
previous version.

在 2013年7月16日星期二UTC+8上午11时39分58秒,Jaguar写道:

Hi,
We are maintaining some daily indexes in multi-nodes cluster. The old
indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?

Thanks!
Jaguar

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for your reply! You are right that there are sub-directory named 1
or even 2. And there were several node reboot along the way.
But I still don't get it, why do those deleted index shows up after reboot.
It seems the lucene index files were not actually deleted during the
"deletion".
Moreover, there were a lot of error complaining missing files or shards in
the master log file, and the master also didn't accept new nodes joining
the cluster. After manually "close" those indexes, the master behaves
normally.
在 2013-7-16 下午12:49,"Ivan Brusic" ivan@brusic.com写道:

You potentially had a split brain at one point and indexed data to two
different clusters. Or you had two instances running on the same server.
You should double check that there is only one directory called 0 under the
data directory.

--
Ivan

On Mon, Jul 15, 2013 at 8:40 PM, Jaguar xiong.jaguar@gmail.com wrote:

Some more detail, this is with version 0.90. Not sure if it's the same
for previous version.

在 2013年7月16日星期二UTC+8上午11时39分58秒,Jaguar写道:

Hi,
We are maintaining some daily indexes in multi-nodes cluster. The
old indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?

Thanks!
Jaguar

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

If you had two instances on the same machine with the same data path, each
instance will create a numbered directory in the
<path.data>/<cluster.name>/nodes
directory. Each directory thinks it has the same index data. When you
deleted the index, only one instance on the machine received the request
and only one of the directories was cleaned up. When a node was restarted,
it used the old directory.

What I would first determine if you truly did have a split brain or two
instances running on the same machine. If so, find a fix for that issue.
Are you using the service wrapper? I found that some of the default timeout
settings are too lenient. You could parse your log files with the lifecycle
option in the es2unix script: https://github.com/elasticsearch/es2unix

--
Ivan

On Mon, Jul 15, 2013 at 10:29 PM, 熊贻青 xiong.jaguar@gmail.com wrote:

Thanks for your reply! You are right that there are sub-directory named 1
or even 2. And there were several node reboot along the way.
But I still don't get it, why do those deleted index shows up after
reboot. It seems the lucene index files were not actually deleted during
the "deletion".
Moreover, there were a lot of error complaining missing files or shards in
the master log file, and the master also didn't accept new nodes joining
the cluster. After manually "close" those indexes, the master behaves
normally.
在 2013-7-16 下午12:49,"Ivan Brusic" ivan@brusic.com写道:

You potentially had a split brain at one point and indexed data to two

different clusters. Or you had two instances running on the same server.
You should double check that there is only one directory called 0 under the
data directory.

--
Ivan

On Mon, Jul 15, 2013 at 8:40 PM, Jaguar xiong.jaguar@gmail.com wrote:

Some more detail, this is with version 0.90. Not sure if it's the same
for previous version.

在 2013年7月16日星期二UTC+8上午11时39分58秒,Jaguar写道:

Hi,
We are maintaining some daily indexes in multi-nodes cluster. The
old indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?

Thanks!
Jaguar

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.