Hi,
We are maintaining some daily indexes in multi-nodes cluster. The old
indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?
Some more detail, this is with version 0.90. Not sure if it's the same for
previous version.
在 2013年7月16日星期二UTC+8上午11时39分58秒，Jaguar写道：
Hi,
We are maintaining some daily indexes in multi-nodes cluster. The old
indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?
You potentially had a split brain at one point and indexed data to two
different clusters. Or you had two instances running on the same server.
You should double check that there is only one directory called 0 under the
data directory.
Some more detail, this is with version 0.90. Not sure if it's the same for
previous version.
在 2013年7月16日星期二UTC+8上午11时39分58秒，Jaguar写道：
Hi,
We are maintaining some daily indexes in multi-nodes cluster. The old
indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?
Thanks for your reply! You are right that there are sub-directory named 1
or even 2. And there were several node reboot along the way.
But I still don't get it, why do those deleted index shows up after reboot.
It seems the lucene index files were not actually deleted during the
"deletion".
Moreover, there were a lot of error complaining missing files or shards in
the master log file, and the master also didn't accept new nodes joining
the cluster. After manually "close" those indexes, the master behaves
normally.
在 2013-7-16 下午12:49，"Ivan Brusic" ivan@brusic.com写道：
You potentially had a split brain at one point and indexed data to two
different clusters. Or you had two instances running on the same server.
You should double check that there is only one directory called 0 under the
data directory.
Some more detail, this is with version 0.90. Not sure if it's the same
for previous version.
在 2013年7月16日星期二UTC+8上午11时39分58秒，Jaguar写道：
Hi,
We are maintaining some daily indexes in multi-nodes cluster. The
old indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?
If you had two instances on the same machine with the same data path, each
instance will create a numbered directory in the
<path.data>/<cluster.name>/nodes
directory. Each directory thinks it has the same index data. When you
deleted the index, only one instance on the machine received the request
and only one of the directories was cleaned up. When a node was restarted,
it used the old directory.
What I would first determine if you truly did have a split brain or two
instances running on the same machine. If so, find a fix for that issue.
Are you using the service wrapper? I found that some of the default timeout
settings are too lenient. You could parse your log files with the lifecycle
option in the es2unix script: GitHub - elastic/es2unix: Command-line ES
Thanks for your reply! You are right that there are sub-directory named 1
or even 2. And there were several node reboot along the way.
But I still don't get it, why do those deleted index shows up after
reboot. It seems the lucene index files were not actually deleted during
the "deletion".
Moreover, there were a lot of error complaining missing files or shards in
the master log file, and the master also didn't accept new nodes joining
the cluster. After manually "close" those indexes, the master behaves
normally.
在 2013-7-16 下午12:49，"Ivan Brusic" ivan@brusic.com写道：
You potentially had a split brain at one point and indexed data to two
different clusters. Or you had two instances running on the same server.
You should double check that there is only one directory called 0 under the
data directory.
Some more detail, this is with version 0.90. Not sure if it's the same
for previous version.
在 2013年7月16日星期二UTC+8上午11时39分58秒，Jaguar写道：
Hi,
We are maintaining some daily indexes in multi-nodes cluster. The
old indexes will be deleted by schedule.
Recently the cluster was restarted, and a different master node was
elected. Quite a few deleted show up again. I check the data directories,
and the actual data are still there.
Is this a known one?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.