During massive delete operation one of the shards (primary) is closed, but marked as ok and cluster was green showing 39 shards of 40 are working. I found what cluster is from logs and see that shard is closed with this error (version 1.7.5):
[2017-02-19 21:16:47,977][WARN ][index.shard ] [Bogeyman] [index] failed to refresh after setting shard to inactive
org.elasticsearch.index.engine.EngineClosedException: [index] CurrentState[CLOSED]
When I didn't find any way to reopen this shard I stopped ES on this node and was hoping that replica shard will be a primary and then cluster will start building a new replica , but it not worked as expected, same issue happened with newly primary shard (former replica) and now I have 2 shards (primary and replica) closed and cluster is yellow, but not red. I don't find any way to recover the cluster and stopped it entirely and started again, that helped, but why this strange shards behaviour is occurs ?