# Failed to recover from translog CurrentState\[CLOSED\]

**URL:** https://discuss.elastic.co/t/failed-to-recover-from-translog-currentstate-closed/192529
**Category:** Elasticsearch
**Created:** [July 27, 2019, 3:40pm UTC](https://discuss.elastic.co/t/failed-to-recover-from-translog-currentstate-closed/192529 "2019-07-27T15:40:18Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)
#### Post date: [July 27, 2019, 3:40pm UTC](https://discuss.elastic.co/t/failed-to-recover-from-translog-currentstate-closed/192529/1 "2019-07-27T15:40:18Z")

</div>

The cluster has unassigned shards possibly due to shard corruption. The process on the node was flaky due to OOM errors and was frequently starting. Disk space was sufficient on the node. I'm yet to verify the hardware failure. But wanted to understand what other events could cause the below. The first error shows unassigned which looks like primary died before replica could recover. Is unexpected error related to networking failure and could it cause translog corruption

```auto
[2019-07-10T04:28:00,129][ERROR][o.e.c.a.s.ShardStateAction] [NgOTVvC] [some_index_2019.07.09][3] unexpected failure while failing shard [shard id [[some_index_2019.07.09][3]], allocation id [rGR2c936RruHuOwdXwKKmQ], primary term [10], message [failed to perform indices:data/write/bulk[s] on replica [some_index_2019.07.09][3], node[YWO9RoPHSXyxPcthgcXILQ], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=rGR2c936RruHuOwdXwKKmQ], unassigned_info[[reason=PRIMARY_FAILED], at[2019-07-10T04:10:11.258Z], delayed=false, details[primary failed while replica initializing], allocation_status[no_attempt]]], failure [NodeDisconnectedException[[YWO9RoP][172.xx.xx.xx:9300][indices:data/write/bulk[s][r]] disconnected]], markAsStale [true]]    

```

```auto
[2019-07-10T08:43:58,732][o.e.i.e.Engine] [c67LMTy] [some_index_2019.07.09][3] failed engine [failed to recover from translog]
    org.elasticsearch.index.engine.EngineException: failed to recover from translog
            at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:406) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:377) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:98) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1297) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:420) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1567) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$5(IndexShard.java:2020) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:635) [elasticsearch-6.3.1.jar:6.3.1]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
    Caused by: org.elasticsearch.index.shard.IllegalIndexShardStateException: CurrentState[CLOSED] operation only allowed when recovering, origin [LOCAL_TRANSLOG_RECOVERY]
            at org.elasticsearch.index.shard.IndexShard.ensureWriteAllowed(IndexShard.java:1444) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:674) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1236) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.shard.IndexShard.runTranslogRecovery(IndexShard.java:1265) ~[elasticsearch-6.3.1.jar:6.3.1]
            at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:404) ~[elasticsearch-6.3.1.jar:6.3.1]
            ... 13 more

```

---

<div class="post-metadata">

### Author: ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)
#### Post date: [July 28, 2019, 5:37pm UTC](https://discuss.elastic.co/t/failed-to-recover-from-translog-currentstate-closed/192529/2 "2019-07-28T17:37:06Z")

</div>

Can someone please take a look I am more than happy to provide more details

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [August 7, 2019, 12:51pm UTC](https://discuss.elastic.co/t/failed-to-recover-from-translog-currentstate-closed/192529/3 "2019-08-07T12:51:09Z")

</div>

This doesn't indicate a corruption, no, this indicates that a primary shard was recovering some missing operations from its local translog and while that recovery was happening the allocation of that primary shard was cancelled. This can happen if the node holding this shard left the cluster and then joined the cluster again while the primary was recovering, which could be because of a network issue.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 4, 2019, 12:51pm UTC](https://discuss.elastic.co/t/failed-to-recover-from-translog-currentstate-closed/192529/4 "2019-09-04T12:51:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
