Hi,
We have an data corruption issue in Elasticsearch. We only have 1 node. We use it for our logstash setup.
The machine crashed and we needed to run an filesystem check (and it repaired a lot of inodes and a lot of other stuff, before we could use it.
When finally started, we see the following error messages:
[2015-07-21 16:05:30,327][WARN ][cluster.action.shard ] [loghost] [logstash-2015.07.21][4] received shard failed for [logstash-2015.07.21][4], node[ps5dpZM3SIaXe-kxPG82Kg], [P], s[INITIALIZING], indexUUID [fJVMIfuKSP6VTxsVuLi0Ag], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[logstash-2015.07.21][4] failed recovery]; nested: EngineCreationFailureException[[logstash-2015.07.21][4] failed to create engine]; nested: NoSuchFileException[/var/lib/elasticsearch//nodes/0/indices/logstash-2015.07.21/4/index/segments_1r]; ]]
[2015-07-21 16:05:40,327][WARN ][indices.cluster ] [loghost] [[logstash-2015.07.21][4]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [logstash-2015.07.21][4] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [logstash-2015.07.21][4] failed to create engine
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:153)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1351)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1346)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:866)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:233)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.nio.file.NoSuchFileException: /var/lib/elasticsearch//nodes/0/indices/logstash-2015.07.21/4/index/segments_1r
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:334)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:172)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:733)
at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:341)
at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:462)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:923)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:769)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:458)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:794)
at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:1134)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:151)
... 9 more
[2015-07-21 16:05:40,331][WARN ][cluster.action.shard ] [loghost] [logstash-2015.07.21][4] received shard failed for [logstash-2015.07.21][4], node[ps5dpZM3SIaXe-kxPG82Kg], [P], s[INITIALIZING], indexUUID [fJVMIfuKSP6VTxsVuLi0Ag], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[logstash-2015.07.21][4] failed recovery]; nested: EngineCreationFailureException[[logstash-2015.07.21][4] failed to create engine]; nested: NoSuchFileException[/var/lib/elasticsearch//nodes/0/indices/logstash-2015.07.21/4/index/segments_1r]; ]]
All shards are UNASSIGNED:
[root@loghost:~]# curl localhost:9200/_cat/shards
logstash-2015.07.21 4 r UNASSIGNED
logstash-2015.07.21 4 p UNASSIGNED
logstash-2015.07.21 0 p UNASSIGNED
logstash-2015.07.21 0 r UNASSIGNED
logstash-2015.07.21 3 p UNASSIGNED
logstash-2015.07.21 3 r UNASSIGNED
logstash-2015.07.21 1 p UNASSIGNED
logstash-2015.07.21 1 r UNASSIGNED
logstash-2015.07.21 2 p UNASSIGNED
logstash-2015.07.21 2 r UNASSIGNED
[root@loghost:~]# curl -XGETs 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
red open kibana-int 5 1
red open logstash-2015.07.16 5 1
red open logstash-2015.07.17 5 1
red open logstash-2015.07.20 5 1
red open logstash-2015.07.19 5 1
red open logstash-2015.07.21 5 1
red open logstash-2015.07.18 5 1
I've tried to look on the www to rebuild indixes, but none of them work.
Anybody have ideas?
CentOS 6, x86_64
Elasticsearch 1.6.0 (6GB jvm XMX)
JVM: jdk1.7.0_75
Kind regards,
Werner