JVM core dump with official Docker image for ElasticSearch 5.5.3

[Andre_Caron (André Caron)]

Hi there!

I'm porting my application to ElasticSearch and Kibana 5.5 (from 1.7 and 4.1, respectively). I'm not migrating any data (yet), only trying to get my application to run against a fresh instance of 5.5.

While doing this, I hit a JVM core dump several times with ElasticSearch 5.5.3 (also occurred on ElasticSearch 5.4.3).

I reported the issue on GitHub (see https://github.com/elastic/elasticsearch-docker/issues/137) with full details. The issue creation tips there suggested sending the info here to reach a wider audience.

Has anyone hit this issue before?

Is there any additional info I can provide that would help shed more light on the issue?

André

[dliappis (Dimitrios Liappis)]

Looking at the docker inspect output in https://github.com/elastic/elasticsearch-docker/issues/137#issue-275738753 it seems that you are using a bind mount for /usr/share/elasticsearch so you could check the local mount for the .log file (probably though this output is after you tried to unsuccessfully bind-mount the whole /usr/share/elasticsearch dir to extract the log file).

At any rate, if you wish to fish out a file from Docker's union filesystem, in your case, since you are using AUFS with boot2docker, you can always jump inside the VM where docker runs and as root:

cd /var/lib/docker/aufs
find ./ -name "*hs_err_pid1.log"

On an Ubuntu 14.04 VM I installed the latest docker-ce, configured it to use aufs by setting:

root@m01:/var/lib/docker/aufs# cat /etc/docker/daemon.json 
{"storage-driver": "aufs"}

and then started Elasticsearch 5.6.3, without any bind mounts or volumes:

docker run -d --name es563 docker.elastic.co/elasticsearch/elasticsearch:5.6.3

Then I entered the container, created a temp file simulating ES's log file and finally killed the container by abruptly ending the ES process:

docker exec -ti es563 /bin/bash
touch hs_err_pid1.log
kill 1

The as the root user in the VM, I can find the log file using:

# cd /var/lib/docker/aufs/
# find ./ -name "*hs_err_pid1.log"
./diff/0dcb78cf0c2262cb667e6c89dbf96216336697fc4710d26cec4bdc7faf354514/usr/share/elasticsearch/hs_err_pid1.log

When you have the contents this file you can paste them here for further troubleshooting.

[Andre_Caron (André Caron)]

Hi @dliappis!

Thanks for the tip. I'll try that and get back here as soon as I have a chance!

[Andre_Caron (André Caron)]

Alright, so I managed to get back to this. I came in this morning with the same crash as last time. This one doesn't have the bind mount.

Core dumps were not enabled, but I still recovered the hs_err_pid1.log file.

I can't seem to upload it here. Attachments seem to be limited to images and the file contents exceed the 7000 character limit on post bodies. I pasted the contents to the GitHub issue to work around these limitations.

Let me know if the log file provides any insight.