ES 7.10
original title: how can I delete a write index which has no allocated shards.
I started the post after struggling with this for a day but, as is often the case, the action of documenting the problem and and double checking the details I ended up solving it.
Having written this I thought someone else may profit from my experience...
I had a disaster with my custer where I lost a lot of shards belonging to over a 100 indexes. This also rendered kibana effectively dead -- everything just timed out.
GIven the kibana was dead I was reduced to using the API directly
I first deleted all indexes that were not green but one failed. That was the write index for a datastream as such is protected by ES. A write index is the current head of a datastream, i.e. the index that is currently being written.
I then tried to delete the datastream but ES insisted that it did not exist. I th
curl --noproxy localhost -X DELETE "localhost:9200//_data_stream/dns_ds"
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index []","resource.type":"index_expression","resource.id":"","index_uuid":"_na_","index":""}],"type":"index_not_found_exception","reason":"no such index []","resource.type":"index_expression","resource.id":"","index_uuid":"_na_","index":""},"status":404}
I then tried listing all datastreams and got the same error.
When I look in the logs of the ES servers I see errors about all shards failed no surprise there
ran GET localhost:9200/_cluster/allocation/explain?pretty and I found each node says
primary shard for this replica is not yet active as the reason that it could not allocate the shard. This has a "catch 22" feel about it!.
I got kibana up by reinstalling and tweaked the config file to tell it to use a new set of indexes (.kibana-new).
I then deleted the data stream and now my cluster is finally GREEN again and I can look seriously seeing what I can restore from snapshots!
In hindsight I should have put my initial efforts into getting kibana up and running
I still don't know the answer to the original question why I could not delete the data stream from the API which was the fundamental problem