Elastic search OOM after running 3 months

asticsearch version is elasticsearch-5.0.0
jdk version is 1.8.0_66
detail exception is following:
i had enable HeapDumpOnOutOfMemoryError, i kept the dump file, it has 447M after compressed.
if any one has interest, i may upload to somewhere netdisk.
``
[2017-01-26T14:51:06,704][WARN ][o.e.i.e.Engine ] [loges13] [weather-log-2017-01-26][4] failed engine [already closed by tragic event on the index writer]
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.index.engine.InternalEngine.failOnTragicEvent(InternalEngine.java:947) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.engine.InternalEngine.maybeFailEngine(InternalEngine.java:968) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:413) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:552) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:542) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnReplica(TransportIndexAction.java:166) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onReplicaShard(TransportShardBulkAction.java:457) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onReplicaShard(TransportShardBulkAction.java:74) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnReplica(TransportWriteAction.java:85) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnReplica(TransportWriteAction.java:50) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:457) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:434) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationLock(IndexShard.java:1667) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationLock(TransportReplicationAction.java:861) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:525) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:418) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:408) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1348) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:504) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.0.jar:5.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]

`

I'd first encourage upgrading both elasticsearch and the jvm.

Then install monitoring to understand more about what is happening.

Also tell us more about your machine, settings, indices, shards...

as now, i do not have a plan to upgrade jvm, because it's on product env.
i may try to upgrade elasticsearch.
i will keep on watch if there are any other exception

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.