org.elasticsearch.index.engine.IndexFailedEngineException

Elastic Stack Elasticsearch
1

Hi Everybody,

I have installed logstash and elasticsearch. Logstash is collecting data for a file and shipping them to elasticsearch to the index myindex. This works fine but as the data collected by logstash increased, elasticsearch keep throwing the following exception

Caused by: org.elasticsearch.index.engine.IndexFailedEngineException: Index failed for [myindex#AVmr5pf6nI-AdjcpoVYx]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:418) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnReplica(TransportIndexAction.java:166) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onReplicaShard(TransportShardBulkAction.java:454) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onReplicaShard(TransportShardBulkAction.java:73) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnReplica(TransportWriteAction.java:81) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnReplica(TransportWriteAction.java:48) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:465) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:438) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationLock(IndexShard.java:1671) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:538) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:422) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:412) ~[elasticsearch-5.1.1.jar:5.1.1]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:168) ~[?:?]
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:176) ~[?:?]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:133) ~[?:?]
at com.floragunn.searchguard.SearchGuardPlugin$2$1.messageReceived(SearchGuardPlugin.java:265) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/nodes/0/indices/ARosIG6LQM6BFYH2una3ZA/1/index/_2.fdx: Read-only file system
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) ~[?:1.8.0_111]
at java.nio.file.Files.newOutputStream(Files.java:216) ~[?:1.8.0_111]
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:413) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:409) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]

Sure, the most important part of this stacktrace is

 Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/nodes/0/indices/ARosIG6LQM6BFYH2una3ZA/1/index/_2.fdx: Read-only file system

But unfortunately for me, I'm not able to understand this message ;).
I'm using elasticsearch 5 and logstash 5.

Any clue please?

2

It looks like something catastrophic happened to your filesystem. For example, a mount was unmounted and remounted as read-only. This is not an Elasticsearch issue, but a problem with your environment.

3

@jasontedor Thanks. My system is been used by many System admins. I guess I will have to talk to all of them to determine who did what?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.