JVM core dump with official Docker image for ElasticSearch 5.5.3

Hi there!

I'm porting my application to ElasticSearch and Kibana 5.5 (from 1.7 and 4.1, respectively). I'm not migrating any data (yet), only trying to get my application to run against a fresh instance of 5.5.

While doing this, I hit a JVM core dump several times with ElasticSearch 5.5.3 (also occurred on ElasticSearch 5.4.3).

I reported the issue on GitHub (see https://github.com/elastic/elasticsearch-docker/issues/137) with full details. The issue creation tips there suggested sending the info here to reach a wider audience.

Has anyone hit this issue before?

Is there any additional info I can provide that would help shed more light on the issue?


Looking at the docker inspect output in https://github.com/elastic/elasticsearch-docker/issues/137#issue-275738753 it seems that you are using a bind mount for /usr/share/elasticsearch so you could check the local mount for the .log file (probably though this output is after you tried to unsuccessfully bind-mount the whole /usr/share/elasticsearch dir to extract the log file).

At any rate, if you wish to fish out a file from Docker's union filesystem, in your case, since you are using AUFS with boot2docker, you can always jump inside the VM where docker runs and as root:

cd /var/lib/docker/aufs
find ./ -name "*hs_err_pid1.log"

On an Ubuntu 14.04 VM I installed the latest docker-ce, configured it to use aufs by setting:

root@m01:/var/lib/docker/aufs# cat /etc/docker/daemon.json 
{"storage-driver": "aufs"}

and then started Elasticsearch 5.6.3, without any bind mounts or volumes:

docker run -d --name es563 docker.elastic.co/elasticsearch/elasticsearch:5.6.3

Then I entered the container, created a temp file simulating ES's log file and finally killed the container by abruptly ending the ES process:

docker exec -ti es563 /bin/bash
touch hs_err_pid1.log
kill 1

The as the root user in the VM, I can find the log file using:

# cd /var/lib/docker/aufs/
# find ./ -name "*hs_err_pid1.log"

When you have the contents this file you can paste them here for further troubleshooting.

Hi @dliappis!

Thanks for the tip. I'll try that and get back here as soon as I have a chance!

Alright, so I managed to get back to this. I came in this morning with the same crash as last time. This one doesn't have the bind mount.

Core dumps were not enabled, but I still recovered the hs_err_pid1.log file.

I can't seem to upload it here. Attachments seem to be limited to images and the file contents exceed the 7000 character limit on post bodies. I pasted the contents to the GitHub issue to work around these limitations.

Let me know if the log file provides any insight.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.