TranslogCorruptedException after restarting ES

Elastic Stack Elasticsearch
1

Hi All,

I am getting IndexShardGatewayRecoveryException after my ES instance got restarted. I am using ES 1.4.4
Can someone help me to fix this?

 [2015-09-14 11:57:50,186][WARN ][indices.cluster          ] [My Node] [failing_index][2] failed to start shard
    org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [failing_index][2] failed to recover shard
    	at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:287)
    	at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    Caused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog corruption while reading from stream
    	at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:70)
    	at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:257)
    	... 4 more
    Caused by: java.io.EOFException
    	at org.elasticsearch.common.io.stream.InputStreamStreamInput.readBytes(InputStreamStreamInput.java:53)
    	at org.elasticsearch.index.translog.BufferedChecksumStreamInput.readBytes(BufferedChecksumStreamInput.java:55)
    	at org.elasticsearch.common.io.stream.StreamInput.readBytesReference(StreamInput.java:86)
    	at org.elasticsearch.common.io.stream.StreamInput.readBytesReference(StreamInput.java:74)
    	at org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:353)
    	at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:68)
    	... 5 more
2

Hi,

Did the disk get full or something happen? On your node look for:

/path/to/esdata/<cluster_name>/nodes/<node_number>/indices/failing_index/2/translog/translog-*

Stop Elasticsearch, then you can try moving the files from inside the translog directory to another location. Then start Elasticsearcha and see if that fixes it.

1 Like