Getting exception - EngineCreationFailureException

Elastic Stack Elasticsearch
1

Hi

I am getting below exceptions repeatedly which is generating multi GBs of log files, and the disk get out of space very quickly.

Caused by: [logstash-2016.09.24][[logstash-2016.09.24][4]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/focom/22000/db/ELASTICSEARCH/nodes/0/indices/logstash-2016.09.24/4/translog/translog-6.ckp];
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:155)
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
	at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515)
	at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499)
	at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:972)
	at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:944)
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
	... 5 more
Caused by: java.nio.file.NoSuchFileException: /focom/22000/db/ELASTICSEARCH/nodes/0/indices/logstash-2016.09.24/4/translog/translog-6.ckp
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
	at java.nio.file.Files.newByteChannel(Files.java:361)
	at java.nio.file.Files.newByteChannel(Files.java:407)
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
	at java.nio.file.Files.newInputStream(Files.java:152)
	at org.elasticsearch.index.translog.Checkpoint.read(Checkpoint.java:82)
	at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:330)
	at org.elasticsearch.index.translog.Translog.<init>(Translog.java:179)
	at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:208)
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:151)
	... 11 more

Below is the cluster health:

{
"cluster_name": "TEST",
"status": "red",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 1691,
"active_shards": 1691,
"relocating_shards": 0,
"initializing_shards": 2,
"unassigned_shards": 1709,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 10,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 2169,
"active_shards_percent_as_number": 49.706055261610814
}

In the index list (_cat/indices), I have seen couple of indices were in RED status.
I am not sure what is the reason behind these repeated logs and how to fix it.
Please help.

ES version - 2.3.2

2

there are several bugs related to this in the 2.x series. I'd recommend to upgrade to 2.4.latest and see if it can recover. If not please report back.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.