How to delete a write index that had no allocated shards

ES 7.10

original title: how can I delete a write index which has no allocated shards.

I started the post after struggling with this for a day but, as is often the case, the action of documenting the problem and and double checking the details I ended up solving it.

Having written this I thought someone else may profit from my experience...

I had a disaster with my custer where I lost a lot of shards belonging to over a 100 indexes. This also rendered kibana effectively dead -- everything just timed out.

GIven the kibana was dead I was reduced to using the API directly

I first deleted all indexes that were not green but one failed. That was the write index for a datastream as such is protected by ES. A write index is the current head of a datastream, i.e. the index that is currently being written.

I then tried to delete the datastream but ES insisted that it did not exist. I th

curl --noproxy localhost -X DELETE "localhost:9200//_data_stream/dns_ds"
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index []","resource.type":"index_expression","":"","index_uuid":"_na_","index":""}],"type":"index_not_found_exception","reason":"no such index []","resource.type":"index_expression","":"","index_uuid":"_na_","index":""},"status":404}

I then tried listing all datastreams and got the same error.

When I look in the logs of the ES servers I see errors about all shards failed no surprise there

ran GET localhost:9200/_cluster/allocation/explain?pretty and I found each node says
primary shard for this replica is not yet active as the reason that it could not allocate the shard. This has a "catch 22" feel about it!.

I got kibana up by reinstalling and tweaked the config file to tell it to use a new set of indexes (.kibana-new).

I then deleted the data stream and now my cluster is finally GREEN again and I can look seriously seeing what I can restore from snapshots!

In hindsight I should have put my initial efforts into getting kibana up and running

I still don't know the answer to the original question why I could not delete the data stream from the API which was the fundamental problem

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.