[logstash-2015.07.21][4] failed to create engine


(Werner Dijkerman) #1

Hi,

We have an data corruption issue in Elasticsearch. We only have 1 node. We use it for our logstash setup.
The machine crashed and we needed to run an filesystem check (and it repaired a lot of inodes and a lot of other stuff, before we could use it.

When finally started, we see the following error messages:
[2015-07-21 16:05:30,327][WARN ][cluster.action.shard ] [loghost] [logstash-2015.07.21][4] received shard failed for [logstash-2015.07.21][4], node[ps5dpZM3SIaXe-kxPG82Kg], [P], s[INITIALIZING], indexUUID [fJVMIfuKSP6VTxsVuLi0Ag], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[logstash-2015.07.21][4] failed recovery]; nested: EngineCreationFailureException[[logstash-2015.07.21][4] failed to create engine]; nested: NoSuchFileException[/var/lib/elasticsearch//nodes/0/indices/logstash-2015.07.21/4/index/segments_1r]; ]]
[2015-07-21 16:05:40,327][WARN ][indices.cluster ] [loghost] [[logstash-2015.07.21][4]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [logstash-2015.07.21][4] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [logstash-2015.07.21][4] failed to create engine
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:153)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1351)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1346)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:866)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:233)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.nio.file.NoSuchFileException: /var/lib/elasticsearch//nodes/0/indices/logstash-2015.07.21/4/index/segments_1r
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:334)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:172)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:733)
at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:341)
at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:462)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:923)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:769)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:458)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:794)
at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:1134)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:151)
... 9 more
[2015-07-21 16:05:40,331][WARN ][cluster.action.shard ] [loghost] [logstash-2015.07.21][4] received shard failed for [logstash-2015.07.21][4], node[ps5dpZM3SIaXe-kxPG82Kg], [P], s[INITIALIZING], indexUUID [fJVMIfuKSP6VTxsVuLi0Ag], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[logstash-2015.07.21][4] failed recovery]; nested: EngineCreationFailureException[[logstash-2015.07.21][4] failed to create engine]; nested: NoSuchFileException[/var/lib/elasticsearch//nodes/0/indices/logstash-2015.07.21/4/index/segments_1r]; ]]

All shards are UNASSIGNED:
[root@loghost:~]# curl localhost:9200/_cat/shards
logstash-2015.07.21 4 r UNASSIGNED
logstash-2015.07.21 4 p UNASSIGNED
logstash-2015.07.21 0 p UNASSIGNED
logstash-2015.07.21 0 r UNASSIGNED
logstash-2015.07.21 3 p UNASSIGNED
logstash-2015.07.21 3 r UNASSIGNED
logstash-2015.07.21 1 p UNASSIGNED
logstash-2015.07.21 1 r UNASSIGNED
logstash-2015.07.21 2 p UNASSIGNED
logstash-2015.07.21 2 r UNASSIGNED

[root@loghost:~]# curl -XGETs 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
red open kibana-int 5 1
red open logstash-2015.07.16 5 1
red open logstash-2015.07.17 5 1
red open logstash-2015.07.20 5 1
red open logstash-2015.07.19 5 1
red open logstash-2015.07.21 5 1
red open logstash-2015.07.18 5 1

I've tried to look on the www to rebuild indixes, but none of them work.

Anybody have ideas?
CentOS 6, x86_64
Elasticsearch 1.6.0 (6GB jvm XMX)
JVM: jdk1.7.0_75

Kind regards,
Werner


(Colin Goodheart-Smithe) #2

It looks like the crash or the file system check corrupted your index files as there are required files missing. Unfortunately without replica shards to copy the data from there is no way to repair your index and retrieve your data. You will need to delete the index and re-index the data from its original source.


(Werner Dijkerman) #3

Bummer..

Thank you for your answer.


(system) #4