ES 6.2.1 Unassigned shards due to DANGLING_INDEX_IMPORTED

[ld_pvl]

Hi Team,

I am getting unassigned shards after I added a new data node to the cluster. This is the sequence of actions that I did prior that lead to unassigned shards:

1. Cluster is green
2. Then on host A, where there is a single node with name node_0 (this node has all master, data, ingest set to true) running, I stopped node_0.
3. On the same host A, I started node_0 back up with a new node, node_1 which is of type data.
4. Cluster now has unassigned shards.
5. Tried to remove node_1 and restarted node_0 to check, still the same amount of unassigned shards.

Upon running http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason&v, the reason for unassigned shards is dangling index imported which is mentioned here https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html along with various other reasons but without a resolution advice. It would be really good if the docs could be updated with some resolution advice for each unassigned cause.

I tried rerouting as follows but did not help - same number of unassigned shards remain:

curl -H 'Content-Type: application/json' -XPOST 'localhost:9200/_cluster/reroute' -d '{
    "commands": [{
        "allocate_stale_primary": {
            "index": "my_index",
            "shard": 1,
            "node": "node_0",
            "accept_data_loss": true
        }
    }]
}'

Previously I used to use the allocate command for rerouting but doesn't look like it's available anymore.

What's the best way to solve the unassigned shards problem caused by dangling index imported?

Thanks a lot,

[JimP (James Peppercorn)]

Did you ever get this resolved? I'm in the same boat, and don't want to lose any data.

[ld_pvl]

I had to clean up stuff to get the cluster back to green state but some data were lost as part of the clean up.

I could afford losing my data though so I did not do the digging too thoroughly. When I looked inside the data folder for the indices, I noticed there were no data but there were some references to those problematic indices and shards which I then (I think) deleted.

[JimP (James Peppercorn)]

Thanks for the quick response!