Elasticsearch 6.4.2 crashes after some time running perfectly fine

[orivera]

Elasticsearch dies after some time running perfectly fine, it leaves no entry in the elasticsearch error but there is JVM error which I uploaded here: https://gist.github.com/ofrivera/ece25bbd7d451c85241ca45517b6b85d

This how my config looks like:
ES:
cluster.name: media-es
node.name: media04
network.host: media04.my.net
http.port: 9200
discovery.zen.ping.unicast.hosts: [media01.my.net,media02.my.net,media03.my.net,media05.my.net]
discovery.zen.minimum_master_nodes: 2
path.data: /elasticsearch/data
path.logs: /var/log/elasticsearch
node.master: true
node.data: true

reindex.remote.whitelist: media.my.net:9200
path.repo: ["/mnt/snapshot"]
xpack.security.enabled: false

JVM:
-Xms7943m
-Xmx7943m
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+DisableExplicitGC
-XX:+AlwaysPreTouch
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-Djdk.io.permissionsUseCanonicalPath=true
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

SPECS:
5 nodes: 2 routing 3 data
16gb RAM memory
8cpus

[warkolm (Mark Walkom)]

Can you please clarify the version number, as there is no 6.4.2.1.

What OS are you on?
Can you upgrade the JVM?

[orivera]

sure thanks for the quick reply!

"version" : {
"number" : "6.4.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "04711c2",
"build_date" : "2018-09-26T13:34:09.098244Z",
"build_snapshot" : false,
"lucene_version" : "7.4.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},

CentOS Linux release 7.4.1708 (Core)
Linux 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

RPM:
elasticsearch-6.4.2.rpm
Details:
elasticsearch-6.4.2-1.noarch

java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

I will try to find whats available in terms of updates for my java version. Is there any recommended version?

[warkolm (Mark Walkom)]

You can see what we support here - https://www.elastic.co/support/matrix#matrix_jvm - but we always recommend the latest.

[orivera]

I'm upgrading java using the latest version from here https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

to:
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

Will let it run for a few days and see how works out