did you run strace with -f? Or even better with -ff / --follow-forks --output-separately ?
a little script:
# cat /tmp/bla.sh
uid=$(id -u elasticsearch)
for pid in $(pgrep -u $uid); do
strace -tt -ff -p $pid -o /tmp/strace-$pid &
done
start this shell script ASAP after starting elasticsearch normally, assuming its running as user=elasticsearch, and it will trace all the child processes.
e.g. for my (standard) 9.1.5 setup I see:
# fgrep -h /root/tmp/.java_pid /tmp/strace-* | sort
22:42:01.950851 stat("/proc/9923/root/tmp/.java_pid9923", 0x7efc72750330) = -1 ENOENT (No such file or directory)
22:42:02.055341 stat("/proc/9923/root/tmp/.java_pid9923", {st_mode=S_IFSOCK|0600, st_size=0, ...}) = 0
22:42:02.055441 stat("/proc/9923/root/tmp/.java_pid9923", {st_mode=S_IFSOCK|0600, st_size=0, ...}) = 0
22:42:02.055882 stat("/proc/9923/root/tmp/.java_pid9923", {st_mode=S_IFSOCK|0600, st_size=0, ...}) = 0
22:42:02.056219 connect(57, {sa_family=AF_UNIX, sun_path="/proc/9923/root/tmp/.java_pid9923"}, 110) = 0
22:42:02.059988 connect(57, {sa_family=AF_UNIX, sun_path="/proc/9923/root/tmp/.java_pid9923"}, 110) = 0
and for the log directory:
# fgrep -h /var/log/elasticsearch /tmp/strace-* | sort | head -10
22:41:44.683063 mkdir("/var/log/elasticsearch", 0777) = -1 EEXIST (File exists)
22:41:44.683596 statx(AT_FDCWD, "/var/log/elasticsearch", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|S_ISGID|0750, stx_size=4096, ...}) = 0
22:41:44.692943 chdir("/var/log/elasticsearch") = 0
22:41:45.473736 getcwd("/var/log/elasticsearch", 4096) = 23
22:41:45.683130 getcwd("/var/log/elasticsearch", 4097) = 23
22:41:49.386476 readlink("/var/log/elasticsearch", 0x7efc7274d800, 1023) = -1 EINVAL (Invalid argument)
22:41:49.386587 readlink("/var/log/elasticsearch/elasticsearch_server.json", 0x7efc7274d800, 1023) = -1 EINVAL (Invalid argument)
22:41:49.386755 stat("/var/log/elasticsearch", {st_mode=S_IFDIR|S_ISGID|0750, st_size=4096, ...}) = 0
22:41:49.387075 mkdir("/var/log/elasticsearch", 0777) = -1 EEXIST (File exists)
22:41:49.387574 statx(AT_FDCWD, "/var/log/elasticsearch", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|S_ISGID|0750, stx_size=4096, ...}) = 0
I notice it actually does a chdir into the log directory, then a getcwd, which obviously works for me here. But in your case?
btw, note the calls to attach to the /proc/9923/root/tmp/.java_pid9923 came well after the syscalls to do with the log directory, which is the order we expected.