# Translog recovery stuck\[ES 6.0\] **URL:** \ **Category:** Elasticsearch\ **Created:** [August 25, 2019, 11:37am UTC](https://discuss.elastic.co/t/translog-recovery-stuck-es-6-0/196658 "2019-08-25T11:37:44Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)\ **Post date:** [August 25, 2019, 11:37am UTC](https://discuss.elastic.co/t/translog-recovery-stuck-es-6-0/196658/1 "2019-08-25T11:37:44Z")
It seems that the shard recovery for replica is stuck in translog recovery loop. The recovery stage for `index` goes through but the `translog` stage gets stuck at 100% with no activity for 30m causing a shard-failed and the loop continues. The cluster stays YELLOW with shard stuck INITAILIZING Options tried so far 1. Flush 2. Synced flush 3. Merge 4. Replica switch from 1 to 0 and back 5. Snapshot restore ```auto curl -s localhost:9200/_cat/recovery?active_only | grep index-a index-a 0 23.8m peer translog 10.xxx.zz.yy RFKcotq 10.xxx.xx.xxx 5r8hFLX n/a n/a 289 289 100.0% 289 41607621033 41607621033 100.0% 41607621033 0 0 100.0% index-a 1 25.1m peer translog 10.xxx.yy.z d4V3wQi 10.xxx.xx.xxx 5r8hFLX n/a n/a 386 386 100.0% 386 41622893547 41622893547 100.0% 41622893547 0 0 100.0% ``` Master logs ```auto [DEBUG][o.e.c.s.MasterService] [1tGegQy] processing [shard-failed[shard id [[index-a][0]], allocation id [MXEBhXqzRSaqlAAnvqLbvw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[index-a][0]: Recovery failed from {j9nurFX}{j9nurFXORuu3HQqe2ejusA}{aG66ze-lRjq7EftXXiDBPA}{10.xxx.xx.xx}{10.xxx.xx.xx:9300}{zone=us-east-1b} into {4YTNoLW}{4YTNoLWWQJOxD26OC_8rQw}{lUVUDg7GQF2FmZM_Mri4_A}{10.xxx.xx.xxx}{10.xxx.xx.xxx:9300}{zone=us-east-1a}]; nested: RemoteTransportException[[j9nurFX][10.xxx.xx.xx:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [315] files with total size of [38.7gb]]; nested: ReceiveTimeoutTransportException[[4YTNoLW][10.xxx.xx.xxx:9300][internal:index/shard/recovery/file_chunk] request_id [686636395] timed out after [900000ms]]; ]]]: took [30s] done publishing updated cluster state (version: 963321, uuid: 5XxUS-74QoWDDPks6yyUug) [2019-08-25T00:31:41,748][WARN][o.e.c.s.MasterService] [1tGegQy] cluster state update task [shard-failed[shard id [[index-a][0]], allocation id [MXEBhXqzRSaqlAAnvqLbvw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[index-a][0]: Recovery failed from {j9nurFX}{j9nurFXORuu3HQqe2ejusA}{aG66ze-lRjq7EftXXiDBPA}{10.xxx.xx.xx}{10.xxx.xx.xx:9300}{zone=us-east-1b} into {4YTNoLW}{4YTNoLWWQJOxD26OC_8rQw}{lUVUDg7GQF2FmZM_Mri4_A}{10.xxx.xx.xxx}{10.xxx.xx.xxx:9300}{zone=us-east-1a}]; nested: RemoteTransportException[[j9nurFX][10.xxx.xx.xx:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [315] files with total size of [38.7gb]]; nested: ReceiveTimeoutTransportException[[4YTNoLW][10.xxx.xx.xxx:9300][internal:index/shard/recovery/file_chunk] request_id [686636395] timed out after [900000ms]]; ]]] took [30s] above the warn threshold of 30s ``` ---
**Author:** ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)\ **Post date:** [September 2, 2019, 2:54pm UTC](https://discuss.elastic.co/t/translog-recovery-stuck-es-6-0/196658/2 "2019-09-02T14:54:16Z")
@DavidTurner any pointers here would be helpful. Is this a known issue and has it been solved already? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 2, 2019, 3:03pm UTC](https://discuss.elastic.co/t/translog-recovery-stuck-es-6-0/196658/3 "2019-09-02T15:03:00Z")
The very few log messages you've provided don't tie in with your description of the problem: `ReceiveTimeoutTransportException[[4YTNoLW][10.xxx.xx.xxx:9300][internal:index/shard/recovery/file_chunk] request_id [686636395] timed out after [900000ms]` This tells us that the recovery failed during the `index` phase. I'm not aware of a known issue, but 6.0 is [EOL](https://www.elastic.co/support/eol). Can you reproduce this on a supported version? Do you see any more useful logs on the recovery source or target nodes? ---
**Author:** ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)\ **Post date:** [September 2, 2019, 5:37pm UTC](https://discuss.elastic.co/t/translog-recovery-stuck-es-6-0/196658/4 "2019-09-02T17:37:02Z")
Thanks @DavidTurner I have not been able to reproduce it so far. Have updated the logs from data nodes, hot threads and recovery stats [here](https://gist.github.com/Bukhtawar/28e86eb53660112eb28f4dfdf1f40915) . Let me know if you need additional info. I'm meanwhile trying to pull up additional relevant logs ---
**Author:** ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)\ **Post date:** [September 2, 2019, 6:46pm UTC](https://discuss.elastic.co/t/translog-recovery-stuck-es-6-0/196658/5 "2019-09-02T18:46:50Z")
The log says `index` phase while I think its probably the same message when `translog recovery` fails too. I haven't verified this though ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 30, 2019, 6:46pm UTC](https://discuss.elastic.co/t/translog-recovery-stuck-es-6-0/196658/6 "2019-09-30T18:46:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.