During massive delete operation one of the shards (primary) is closed, but marked as ok and cluster was green showing 39 shards of 40 are working. I found what cluster is from logs and see that shard[14] is closed with this error (version 1.7.5):
[2017-02-19 21:16:47,977][WARN ][index.shard ] [Bogeyman] [index][14] failed to refresh after setting shard to inactive
org.elasticsearch.index.engine.EngineClosedException: [index][14] CurrentState[CLOSED]
at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:289)
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:565)
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:594)
at org.elasticsearch.index.shard.IndexShard.updateBufferSize(IndexShard.java:1079)
at org.elasticsearch.index.shard.IndexShard.markAsInactive(IndexShard.java:1091)
at org.elasticsearch.indices.memory.IndexingMemoryController.markShardAsInactive(IndexingMemoryController.java:405)
at org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:297)
at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:508)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
When I didn't find any way to reopen this shard I stopped ES on this node and was hoping that replica shard will be a primary and then cluster will start building a new replica , but it not worked as expected, same issue happened with newly primary shard (former replica) and now I have 2 shards (primary and replica) closed and cluster is yellow, but not red. I don't find any way to recover the cluster and stopped it entirely and started again, that helped, but why this strange shards behaviour is occurs ?
Thanks