Docker Heap Dump

Elastic Stack Elasticsearch
1

:wave:, howdy!

How do you grab an ES Docker heap dump? From basic setup:

$ docker run -it -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.17.0
$ docker exec -it CONTAINER_ID bash

Trying close to the ECK heap dump guide

root@CONTAINER_ID:/# /usr/share/elasticsearch/jdk/bin/jmap -dump:format=b,file=heap.hprof $(pgrep java)

Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /proc/8/root/tmp/.java_pid8: target process 8 doesn't respond within 10500ms or HotSpot VM not loaded
	at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:104)
	at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
	at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
	at jdk.jcmd/sun.tools.jmap.JMap.executeCommandForPid(JMap.java:128)
	at jdk.jcmd/sun.tools.jmap.JMap.dump(JMap.java:248)
	at jdk.jcmd/sun.tools.jmap.JMap.main(JMap.java:114)

This google returns that I'm running as the wrong user. So checking

root@CONTAINER_ID:/# whoami
root

root@CONTAINER_ID:/# ps -ef | grep java | awk '{print($1,"\t",$2, "\t", $8)}'
elastic+ 	 7 	 /usr/share/elasticsearch/jdk/bin/java
root 	 289 	 grep

However, attempting variations of switcher user su, I'm unable to get the right user to take a heap dump.

root@CONTAINER_ID:/# su - elastic+
su: user elastic+ does not exist
root@CONTAINER_ID:/# su "elastic+"
su: user elastic+ does not exist
root@CONTAINER_ID:/# su - elastic
su: user elastic does not exist

AFAICT from this Stackoverflow, there's no alternative to shifting to the correct user to gain access to capturing the heap dump successfully. (Similar Discuss ballpark, but not resolving: Discuss#171363. No error related ES issues.)

I am expecting a workflow similar to this Medium, but have gotten myself caught on this elastic+ user. Can someone tell me how to run this heap dump successfully? TIA! :pray:

2

I had a hint forward that elastic+ is a shortened username.

$ docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED          STATUS          PORTS                                            NAMES
74b698da5ea3   docker.elastic.co/elasticsearch/elasticsearch:7.17.4   "/bin/tini -- /usr/lā€¦"   17 minutes ago   Up 17 minutes   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   confident_ritchie
$ docker exec -it 74b698da5ea3 bash
> root@74b698da5ea3:/usr/share/elasticsearch# cat /etc/passwd | grep -E 'root|elastic'
root:x:0:0:root:/root:/bin/bash
elasticsearch:x:1000:1000:,,,:/usr/share/elasticsearch:/bin/bash
> root@74b698da5ea3:/usr/share/elasticsearch# su elasticsearch
> elasticsearch@74b698da5ea3:~$ pgrep java
8
> elasticsearch@74b698da5ea3:~$ ./jdk/bin/jmap -dump:live,format=b,file=heap.hprof 8
Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /proc/8/root/tmp/.java_pid8: target process 8 doesn't respond within 10500ms or HotSpot VM not loaded
    at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:105)
    at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
    at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
    at jdk.jcmd/sun.tools.jmap.JMap.executeCommandForPid(JMap.java:127)
    at jdk.jcmd/sun.tools.jmap.JMap.dump(JMap.java:243)
    at jdk.jcmd/sun.tools.jmap.JMap.main(JMap.java:114)

Still erred but now we know more about the permission group we actually need on docker exec:

$ docker exec -u 1000:0 -it 74b698da5ea3 bash
elasticsearch@74b698da5ea3:~$ ./jdk/bin/jmap -dump:format=b,file=heap.hprof 8
Dumping heap to /usr/share/elasticsearch/heap.hprof ...
Heap dump file created [1033192790 bytes in 2.171 secs]
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.