# Deleted index shows up after cluster restart **URL:** \ **Category:** Elasticsearch\ **Created:** [July 16, 2013, 3:39am UTC](https://discuss.elastic.co/t/deleted-index-shows-up-after-cluster-restart/12801 "2013-07-16T03:39:58Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Jaguar](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@Jaguar](https://discuss.elastic.co/u/Jaguar)\ **Post date:** [July 16, 2013, 3:39am UTC](https://discuss.elastic.co/t/deleted-index-shows-up-after-cluster-restart/12801/1 "2013-07-16T03:39:58Z")
Hi, We are maintaining some daily indexes in multi-nodes cluster. The old indexes will be deleted by schedule. Recently the cluster was restarted, and a different master node was elected. Quite a few _deleted_ show up again. I check the data directories, and the actual data are still there. Is this a known one? Thanks! Jaguar -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jaguar](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@Jaguar](https://discuss.elastic.co/u/Jaguar)\ **Post date:** [July 16, 2013, 3:40am UTC](https://discuss.elastic.co/t/deleted-index-shows-up-after-cluster-restart/12801/2 "2013-07-16T03:40:59Z")
Some more detail, this is with version 0.90. Not sure if it's the same for previous version. 在 2013年7月16日星期二UTC+8上午11时39分58秒，Jaguar写道： > Hi, > We are maintaining some daily indexes in multi-nodes cluster. The old > indexes will be deleted by schedule. > Recently the cluster was restarted, and a different master node was > elected. Quite a few _deleted_ show up again. I check the data directories, > and the actual data are still there. > Is this a known one? > > Thanks! > Jaguar -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [July 16, 2013, 4:49am UTC](https://discuss.elastic.co/t/deleted-index-shows-up-after-cluster-restart/12801/3 "2013-07-16T04:49:53Z")
You potentially had a split brain at one point and indexed data to two different clusters. Or you had two instances running on the same server. You should double check that there is only one directory called 0 under the data directory. -- Ivan On Mon, Jul 15, 2013 at 8:40 PM, Jaguar [xiong.jaguar@gmail.com](mailto:xiong.jaguar@gmail.com) wrote: > Some more detail, this is with version 0.90. Not sure if it's the same for > previous version. > > 在 2013年7月16日星期二UTC+8上午11时39分58秒，Jaguar写道： > > > Hi, > > We are maintaining some daily indexes in multi-nodes cluster. The old > > indexes will be deleted by schedule. > > Recently the cluster was restarted, and a different master node was > > elected. Quite a few _deleted_ show up again. I check the data directories, > > and the actual data are still there. > > Is this a known one? > > > > Thanks! > > Jaguar > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jaguar](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@Jaguar](https://discuss.elastic.co/u/Jaguar)\ **Post date:** [July 16, 2013, 5:29am UTC](https://discuss.elastic.co/t/deleted-index-shows-up-after-cluster-restart/12801/4 "2013-07-16T05:29:46Z")
Thanks for your reply! You are right that there are sub-directory named 1 or even 2. And there were several node reboot along the way. But I still don't get it, why do those deleted index shows up after reboot. It seems the lucene index files were not actually deleted during the "deletion". Moreover, there were a lot of error complaining missing files or shards in the master log file, and the master also didn't accept new nodes joining the cluster. After manually "close" those indexes, the master behaves normally. 在 2013-7-16 下午12:49，"Ivan Brusic" [ivan@brusic.com](mailto:ivan@brusic.com)写道： > You potentially had a split brain at one point and indexed data to two > different clusters. Or you had two instances running on the same server. > You should double check that there is only one directory called 0 under the > data directory. > > -- > Ivan > > On Mon, Jul 15, 2013 at 8:40 PM, Jaguar [xiong.jaguar@gmail.com](mailto:xiong.jaguar@gmail.com) wrote: > > > Some more detail, this is with version 0.90. Not sure if it's the same > > for previous version. > > > > 在 2013年7月16日星期二UTC+8上午11时39分58秒，Jaguar写道： > > > > > Hi, > > > We are maintaining some daily indexes in multi-nodes cluster. The > > > old indexes will be deleted by schedule. > > > Recently the cluster was restarted, and a different master node was > > > elected. Quite a few _deleted_ show up again. I check the data directories, > > > and the actual data are still there. > > > Is this a known one? > > > > > > Thanks! > > > Jaguar > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [July 16, 2013, 4:20pm UTC](https://discuss.elastic.co/t/deleted-index-shows-up-after-cluster-restart/12801/5 "2013-07-16T16:20:44Z")
If you had two instances on the same machine with the same data path, each instance will create a numbered directory in the \/\/nodes directory. Each directory thinks it has the same index data. When you deleted the index, only one instance on the machine received the request and only one of the directories was cleaned up. When a node was restarted, it used the old directory. What I would first determine if you truly did have a split brain or two instances running on the same machine. If so, find a fix for that issue. Are you using the service wrapper? I found that some of the default timeout settings are too lenient. You could parse your log files with the lifecycle option in the es2unix script: [GitHub - elastic/es2unix: Command-line ES](https://github.com/elasticsearch/es2unix) -- Ivan On Mon, Jul 15, 2013 at 10:29 PM, 熊贻青 [xiong.jaguar@gmail.com](mailto:xiong.jaguar@gmail.com) wrote: > Thanks for your reply! You are right that there are sub-directory named 1 > or even 2. And there were several node reboot along the way. > But I still don't get it, why do those deleted index shows up after > reboot. It seems the lucene index files were not actually deleted during > the "deletion". > Moreover, there were a lot of error complaining missing files or shards in > the master log file, and the master also didn't accept new nodes joining > the cluster. After manually "close" those indexes, the master behaves > normally. > 在 2013-7-16 下午12:49，"Ivan Brusic" [ivan@brusic.com](mailto:ivan@brusic.com)写道： > > You potentially had a split brain at one point and indexed data to two > > > different clusters. Or you had two instances running on the same server. > > You should double check that there is only one directory called 0 under the > > data directory. > > > > -- > > Ivan > > > > On Mon, Jul 15, 2013 at 8:40 PM, Jaguar [xiong.jaguar@gmail.com](mailto:xiong.jaguar@gmail.com) wrote: > > > > > Some more detail, this is with version 0.90. Not sure if it's the same > > > for previous version. > > > > > > 在 2013年7月16日星期二UTC+8上午11时39分58秒，Jaguar写道： > > > > > > > Hi, > > > > We are maintaining some daily indexes in multi-nodes cluster. The > > > > old indexes will be deleted by schedule. > > > > Recently the cluster was restarted, and a different master node was > > > > elected. Quite a few _deleted_ show up again. I check the data directories, > > > > and the actual data are still there. > > > > Is this a known one? > > > > > > > > Thanks! > > > > Jaguar > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:26am UTC](https://discuss.elastic.co/t/deleted-index-shows-up-after-cluster-restart/12801/6 "2017-07-06T02:26:09Z")