Knowledge dumping here…Lucky for you all this will likely be my last post on this for a while.
The strace of the ls -al command on the dir on which the volume is mounted shows something interesting. The lgetxattr and getxattr syscalls on it show -1 EOPNOTSUPP (Operation not supported) on the volume that causes an issue, and -1 ENODATA (No data available) on the volume that has no issue. Keep in mind that the ls -al command appears to work just fine in both environments, meaning it shows the right output. See [1] for more details. My favorite LLM tells me that this EOPNOTSUPP error just indicates that the underlying filesystem doesn't implement these extended attributes - NOT that there's a problem and this by itself shouldn't be a cause for the JVM to not be able to attach.
The strace output between an elasticsearch server startup that fails vs. one that doesn't did NOT produce anything really all that useful, unless I'm just using it incorrectly (I just did strace <java command I got from 'ps' output>). The syscalls were identical all the way through to where it printed out "Bootstrapping Entitlements" after which the program execution diverged as expected. See [2] for small snippet of results.
I also played around with the EFS CSI driver and storage class that I've been using. ReadWriteMany is not the problem here as it also fails with ReadWriteOnce. Setting directoryPerms=777 in the storage class and adding supplementalGroups on the pod that correspond to the CSI driver's provided gid also doesn't work. In short, it kinda looks like almost all EFS usage is borked unless you explicitly set the uid/gid in the storageclass.
[1] (some paths altered to remove sensitive info)
strace of the ls -al command on the dir on which the problematic efs-created volume is mounted
statx(AT_FDCWD, "/usr/local/mydir/myapp/logs", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|0700, stx_size=6144, ...}) = 0
lgetxattr("/usr/local/mydir/myapp/logs", "security.selinux", 0x5604c91e4340, 255) = -1 EOPNOTSUPP (Operation not supported)
getxattr("/usr/local/mydir/myapp/logs", "system.posix_acl_access", NULL, 0) = -1 EOPNOTSUPP (Operation not supported)
strace of the ls -al command on the dir on which the problem-free ebs-created volume is mounted
statx(AT_FDCWD, "/usr/local/mydir/myapp/logs", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|S_ISGID|0775, stx_size=4096, ...}) = 0
lgetxattr("/usr/local/mydir/myapp/logs", "security.selinux", 0x56016ab0f340, 255) = -1 ENODATA (No data available)
getxattr("/usr/local/mydir/myapp/logs", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available)
getxattr("/usr/local/mydir/myapp/logs", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available)
[2]: strace output of elasticsearch process (some paths altered to remove sensitive info)
access("/usr/local/mydir/myapp/es-server/elasticsearch/lib/cli-launcher/", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/mydir/myapp/es-server/elasticsearch/lib/cli-launcher/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|S_ISGID|0755, st_size=6, ...}) = 0
getdents64(3, 0x55d4e842f540 / 3 entries /, 32768) = 96
getdents64(3, 0x55d4e842f540 / 0 entries /, 32768) = 0
close(3) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7fa10b334b50, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7fa10b2ebc30}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fa109d77000
rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7fa109e76910, parent_tid=0x7fa109e76910, exit_signal=0, stack=0x7fa109d77000, stack_size=0xfef00, tls=0x7fa109e76640}, 88) = -1 ENOSYS (Function not implemented)
clone(child_stack=0x7fa109e75ef0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[34], tls=0x7fa109e76640, child_tidptr=0x7fa109e76910) = 34
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
futex(0x7fa109e76910, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 34, NULL, FUTEX_BITSET_MATCH_ANY[2025-10-23T18:53:31,246][INFO ][org.elasticsearch.bootstrap.Elasticsearch] [myapp-k8s-jbuhc-es-server-0.myapp-k8s-jbuhc-es-server-headless] version[8.19.4], pid[94], build[zip/aa0a7826e719b392e7782716b323c4fb8fa3b392/2025-09-16T22:06:03.940754111Z], OS[Linux/5.15.193-134.215.amzn2.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.16/17.0.16+8]
[2025-10-23T18:53:31,248][INFO ][org.elasticsearch.bootstrap.Elasticsearch] [myapp-k8s-jbuhc-es-server-0.myapp-k8s-jbuhc-es-server-headless] JVM home [/usr/local/mydir/myapp/java], using bundled JDK [false]
[2025-10-23T18:53:31,248][INFO ][org.elasticsearch.bootstrap.Elasticsearch] [myapp-k8s-jbuhc-es-server-0.myapp-k8s-jbuhc-es-server-headless] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, -Dorg.apache.lucene.vectorization.upperJavaFeatureVersion=17, -Des.path.home=/usr/local/mydir/myapp/es-server/elasticsearch, -Des.distribution.type=zip, -Des.java.type=ES_JAVA_HOME, -XX:ReplayDataFile=logs/replay_pid%p.log, -Des.entitlements.enabled=true, -XX:+EnableDynamicAgentLoading, -Djdk.attach.allowAttachSelf=true, --patch-module=java.base=/usr/local/mydir/myapp/es-server/elasticsearch/lib/entitlement-bridge/elasticsearch-entitlement-bridge-8.19.4.jar, --add-exports=java.base/org.elasticsearch.entitlement.bridge=org.elasticsearch.entitlement,java.logging,java.net.http,java.naming,jdk.net, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-4448555691991616784, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:ErrorFile=hs_err_pid%p.log, -Xlog:gc,gc+age=trace,safepoint:file=gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms2048m, -Xmx2048m, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, --module-path=/usr/local/mydir/myapp/es-server/elasticsearch/lib, --add-modules=jdk.net, --add-modules=jdk.management.agent, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
[2025-10-23T18:53:31,248][INFO ][org.elasticsearch.bootstrap.Elasticsearch] [myapp-k8s-jbuhc-es-server-0.myapp-k8s-jbuhc-es-server-headless] Default Locale [en]
[2025-10-23T18:53:31,610][INFO ][org.elasticsearch.nativeaccess.NativeAccess] [myapp-k8s-jbuhc-es-server-0.myapp-k8s-jbuhc-es-server-headless] Using [jna] native provider and native methods for [Linux]
[2025-10-23T18:53:31,733][INFO ][org.elasticsearch.bootstrap.Elasticsearch] [myapp-k8s-jbuhc-es-server-0.myapp-k8s-jbuhc-es-server-headless] Bootstrapping Entitlements