{index_name}/{shard_id}/_state directory is missing and {index_name}/{shard_id}/index directory is empty (all index files are gone) after a few nodes were restarted at the same time

Elastic Stack Elasticsearch
1

Hi,
I am using ElasticSearch version 1.5.0.

Our ElasticSearch cluster has 8 shards with 1 replica, hosted on 7 data
nodes + 1 master node with setting below:

gateway.recover_after_nodes: 4
gateway.recover_after_time: 5m
gateway.expected_nodes: 6

After a few nodes were restarted at the same time, 2 out of 7 shards (both
Primary and Replicat) are in UNASSIGNED status.
Here are my findings when I check the shards data folder for these 2
UNASSIGNED shards:

  1. {index_name}/{shard_id} only exists in 1 out of 7 data nodes. It should
    exist in 2 out of 7 data nodes since we have 1 replica setting.
  2. For {index_name}/{shard_id} which exists in one of the data nodes,
    {index_name}/{shard_id}/_state directory is missing
  3. For {index_name}/{shard_id} which exists in one of the data nodes,
    {index_name}/{shard_id}/index directory has 0 files -> all index files are
    missing.

I am able to simulate this scenario after a few rounds of stopping and
starting the ElasticSearch cluster nodes (a few nodes at a time).
Is this a known issue that {index_name}/{shard_id}/_state directory and the
index files will be lost if the nodes are restarted during
initializing/relocating?

Thanks.

Lee Chuen

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef335410-9d64-4da0-9ce9-052bc6941af9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.