i have a elasticsearch 5.1 on centos 6.8. every day a large index will corrupt ,with error :
[2017-01-17T22:53:15,905][WARN ][o.e.i.e.Engine ] [es01] [logstash-apilog-2017.01.17][3] failed engine [recovery]
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=3vfyl9 actual=qrd7bb (resource=name [_dn9_Lucene54_0.dvd], length [1324210527], checksum [3vfyl9], writtenBy [6.3.0]) (resource=VerifyingIndexOutput(_dn9_Lucene54_0.dvd))
at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1149) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1128) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1157) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:424) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:501) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:474) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) [elasticsearch-5.1.1.jar:5.1.1]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:168) [search-guard-ssl-5.1.1-19.jar:5.1.1-19]
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:176) [search-guard-5-5.1.1-10.jar:?]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:133) [search-guard-ssl-5.1.1-19.jar:5.1.1-19]
at com.floragunn.searchguard.SearchGuardPlugin$2$1.messageReceived(SearchGuardPlugin.java:266) [search-guard-5-5.1.1-10.jar:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
the following is index shards stats
logstash-apilog-2017.01.17 3 p UNASSIGNED
logstash-apilog-2017.01.17 3 r UNASSIGNED
logstash-apilog-2017.01.17 1 p STARTED 21603723 13.1gb 172.16.12.183 es03
logstash-apilog-2017.01.17 1 r STARTED 21603723 13.1gb 172.16.39.123 es02
logstash-apilog-2017.01.17 4 p STARTED 21605111 13.1gb 172.16.12.183 es03
logstash-apilog-2017.01.17 4 r STARTED 21605111 13.1gb 172.16.39.123 es02
logstash-apilog-2017.01.17 2 p STARTED 21600451 13.1gb 172.16.12.183 es03
logstash-apilog-2017.01.17 2 r STARTED 21600451 13.1gb 172.16.38.126 es01
logstash-apilog-2017.01.17 0 r STARTED 21605967 13.1gb 172.16.38.126 es01
logstash-apilog-2017.01.17 0 p STARTED 21605967 13.1gb 172.16.39.123 es02