6.5.0 JVM Crash

Elastic Stack Elasticsearch
1

Hi. Running Elasticsearch 6.5.0 in Docker. Been upgrading our environment and so replacing Elastic servers. Everything was going fine until one of our servers started JVM crashing.

Here's the JVM output: https://gist.github.com/bitva77/d8e47b894263525fcc6fcb8c2755eb90

Running in Docker (18.0.6)

Really have no idea what to do about this.

Any insight would be appreciated.

Thanks!

-Matt

2

Whether sharding was enabled or not, ES still crashed and would loop (docker restarts the container but it crashes again). Server reboot had no effect.

Red Hat 7.5.

I've since destroyed that server and have built a new one, identical and so far so good (knocking on wood).

~6TB cluster size

3 Master
2 Data

It's the 3rd master that crashed. I have the new one in and the cluster is currently doing it's replication thing.

Will update this thread if things go sound again

3

What is the heap size of your data nodes? How many indices/shards do you have in the cluster?

4

26GB heap with 64GB RAM.

239 indices plus 58 "." indices (so 297 total). There are like 30 closed.

4,880 shards.

The number of shards will go down a bit. We were at like 10*2 across the board but have since changed the template to either 4 or 8 *2.

Thanks!

5

Got the JVM crashing again, in a loop:

6

No matter what I try it crashes now: OpenJDK, OracleJDK, docker image, no docker image:

Anything else you need. I guess nobody cares until we pay ya'll money (which we're working on) but any help you can provide now would be great.

7

Are using our official docker image? How much resources are you giving the image?

8

Both actually. And once with the tar.gz file....

9

Oh sorry, resource wise:

26GB heap on a 64GB server with 8 cores

10

Happens with the vanilla image as well. This is Red Hat 7.6.

All I'm running is:

docker run docker.elastic.co/elasticsearch/elasticsearch:6.5.4

it sits for about a minute before crashing

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.