On 2 identical installations of elastic search (5.6.1, single instance, 7 GB of HEAP, 16GB Memory, 8 core) i got a DocumentMissingException from time to time
the setup is
redis -> logstash -> elasticsearch
recently, one of the machines had lots of DocumentMissingExceptions in the log
how can i debug this? what is causing this exceptions? might this be a problem with storage?
Adding Memory + CPU was no solution
here's one exception for example:
[2017-12-04T09:46:48,956][DEBUG][o.e.a.b.TransportShardBulkAction] [node-1] [default-correlation-calls-2017.12.04][0] failed to execute bulk item (update) BulkShardRequest [[default-correlation-calls-2017.12.04][0]] containing [121] requests
org.elasticsearch.index.engine.DocumentMissingException: [call][1512368178:134348487]: document missing
at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:93) ~[elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:82) ~[elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequest(TransportShardBulkAction.java:277) ~[elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:162) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:115) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:70) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) [elasticsearch-5.6.1.jar:5.6.1]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) [elasticsearch-5.6.1.jar:5.6.1]
at
...