Cluster State Red - All Shards Were Deleted but Marvel Says 'Unassigned'

Elastic Stack Elasticsearch
1

I've been having an issue where my one client node just deletes the primary index.. I'm basically discussing it with myself here, as no one has really offered much help.

Now I'm in a position where my cluster state has been 'Red' for hours due to these nonexistent shards. As seen in this Filebeat index, there are no shards available, yet it still wants to think that the index there.

I look for unassigned shards and find absolutely nothing, it doesn't even say '0' shards.

Pasted image1261×524 90.8 KB

I've tried using Curator to delete these but Curator also says the index doesn't exist. Am I stuck just removing and reinstalling Marvel?

2

This somehow resolved itself.

3

Hi Seth,

It looks like this all happened almost exactly at 16:00. I wonder if you all are using a Curator job to delete indices? If so, it's possible that one of the data nodes was not around when that occurred, so it rejoined with "dangling indices" and caused the master to think that it's missing shards when in fact it should just delete those too.

Later, it would be reasonable that Curator would run a second time and delete the dangling shards, which fixed the problem.

Having said that, I noticed the screenshot is showing dates that are probably too soon to be deleted by Curator, so I'll take a gander at the linked discussion and see what's going on over there.

1 Like