Primary shard is closed but still marked as ok and cluster was green

During massive delete operation one of the shards (primary) is closed, but marked as ok and cluster was green showing 39 shards of 40 are working. I found what cluster is from logs and see that shard[14] is closed with this error (version 1.7.5):

[2017-02-19 21:16:47,977][WARN ][index.shard ] [Bogeyman] [index][14] failed to refresh after setting shard to inactive
org.elasticsearch.index.engine.EngineClosedException: [index][14] CurrentState[CLOSED]
at org.elasticsearch.index.engine.Engine.ensureOpen(
at org.elasticsearch.index.engine.InternalEngine.refresh(
at org.elasticsearch.index.shard.IndexShard.refresh(
at org.elasticsearch.index.shard.IndexShard.updateBufferSize(
at org.elasticsearch.index.shard.IndexShard.markAsInactive(
at org.elasticsearch.indices.memory.IndexingMemoryController.markShardAsInactive(
at org.elasticsearch.indices.memory.IndexingMemoryController$
at org.elasticsearch.threadpool.ThreadPool$
at java.util.concurrent.Executors$
at java.util.concurrent.FutureTask.runAndReset(
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(
at java.util.concurrent.ScheduledThreadPoolExecutor$
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$

When I didn't find any way to reopen this shard I stopped ES on this node and was hoping that replica shard will be a primary and then cluster will start building a new replica , but it not worked as expected, same issue happened with newly primary shard (former replica) and now I have 2 shards (primary and replica) closed and cluster is yellow, but not red. I don't find any way to recover the cluster and stopped it entirely and started again, that helped, but why this strange shards behaviour is occurs ?


What version?

It's 1.7.5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.