Hi
I am getting below exceptions repeatedly which is generating multi GBs of log files, and the disk get out of space very quickly.
Caused by: [logstash-2016.09.24][[logstash-2016.09.24][4]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/focom/22000/db/ELASTICSEARCH/nodes/0/indices/logstash-2016.09.24/4/translog/translog-6.ckp];
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:155)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:972)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:944)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
... 5 more
Caused by: java.nio.file.NoSuchFileException: /focom/22000/db/ELASTICSEARCH/nodes/0/indices/logstash-2016.09.24/4/translog/translog-6.ckp
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.index.translog.Checkpoint.read(Checkpoint.java:82)
at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:330)
at org.elasticsearch.index.translog.Translog.<init>(Translog.java:179)
at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:208)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:151)
... 11 more
Below is the cluster health:
{
"cluster_name": "TEST",
"status": "red",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 1691,
"active_shards": 1691,
"relocating_shards": 0,
"initializing_shards": 2,
"unassigned_shards": 1709,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 10,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 2169,
"active_shards_percent_as_number": 49.706055261610814
}
In the index list (_cat/indices), I have seen couple of indices were in RED status.
I am not sure what is the reason behind these repeated logs and how to fix it.
Please help.
ES version - 2.3.2