EntitlementBootstrap failure preventing startup: AttachNotSupportedException: Unable to open socket file

[RainTown (Kevin Maguire)]

The readlinks are IMO irrelevant, it is just checking if paths are symlinks or regular files/dirs. In some scenarios, if it’s expecting the latter, but finds the former, it can be suspicious.

@buitcj

Failing system (looks like part of the way down, those writes are the stack trace that's printed out)

	18:03:05.778196 openat(AT_FDCWD, "/proc/303/cwd/.attach_pid303", O_RDWR|O_CREAT|O_EXCL, 0666) = 44                                
	18:03:05.784629 readlink("/usr/local/mydir/myapp/logs/myapp/.attach_pid303", 0x7f6eaabc1a30, 1023) = -1 EINVAL (Invalid argu ment)                                                                                                                             
	18:03:05.790869 write(1, "\tat sun.tools.attach.VirtualMach"..., 94) = 94                                                         
	18:03:05.790969 write(1, "\tat sun.tools.attach.AttachProvi"..., 108) = 108                                                       
	18:03:05.791072 write(1, "\tat com.sun.tools.attach.Virtual"..., 91) = 91                                                         
	18:03:05.785024 stat(".attach_pid303", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0      

Working system

	17:47:48.890508 openat(AT_FDCWD, "/proc/290/cwd/.attach_pid290", O_RDWR|O_CREAT|O_EXCL, 0666) = 44                               
	17:47:48.891237 readlink("/usr/local/mydir/myapp/logs/myapp/.attach_pid290", 0x7f3a94feba30, 1023) = -1 EINVAL (Invalid argu ment)                                                                                                                             
	17:47:48.991963 unlink("/usr/local/mydir/myapp/logs/myapp/.attach_pid290") = 0                                           

Those logs for failing system dont include the unlink call on /usr/local/mydir/myapp/logs/myapp/.attach_pid290, did you not include it or is it not there.

The logic is a bit confusing - it creates this empty .attach_pid*file but seems to just do a stat on it (in a separate thread) and then unlink it. On a working system. It used to do this elsewhere on the filesystem, but in 8.19.5 (likely all 8.19+) it does so in path.logs directory which it has chdir-ed to so is the CWD.

In a normal shell, as the elasticsearch process running user, can you chdir to the “failing path.logs directory, and while there create then stat then remove an empty file?

–

EDIT: I read a bit, and it seems this “create .attach_pid, stat that file in another thread, then unlink it” sequence is normal. To quote:

Every HotSpot JVM periodically checks for a file named .attach_pid in /tmp or under its /proc//root/tmp. If it finds one, it knows someone wants to attach.

It’s still not clear why thats happening in the directory from path.logs and not one of the temporary directories above. And maybe, in some cases, this mechanism is failing to actually wake up the “Attach Listener”, so your actual attach fails, via the .java_pid* file, fails. Can maybe workaround with “-Dcom.sun.management.jmxremote” to the JVM options as this forces the “Attach Listener” to start without the need for .attach_pid “prompt”.