Hoping David or another Elasticsearch dev/insider can help with this question. Please don’t interpret it as attacking the code - I just want to understand the entitlement check better at the moment. When I look at v8.19.4 of the repo, I don’t find any other place where loadAgent is being called. Who else is (other than the entitlement check) is trying to connect to the JVM? What does this check accomplish and why does Elasticsearch do this in 8.19.4 but not 8.18?
I read there is no way to disable entitlement checks now. Is that accurate?
I am also very curious too, this is one of the more interesting threads at present, so I also hope someone answers.
At least to say "yea, you are right, path.logs can cause this startup error, and that looks like a bug", which I what I would hope. Or "well, sorry, that wont work anymore, but we never guaranteed it would work, see ". Or some variant, or something else entirely.
But, and dont take this as criticism, even if there is an answer, the chances of the explanation helping anything other than you/our understanding seems rather slim.
I personally (not k8s expert) dont really understand (nor obviously do I need to):
But it seems a little in the direction of a "creative workaround" that just happened to work. FYI I had a go at reproducing with docker, but though I could get it to fail to startup, but not to fail in same way you have hit.
This may be helpful to read, since 8.19.4 ships with JDK 24, so ...
I don’t think anyone else is trying to connect back to the JVM, it’s just the entitlements subsystem injecting itself as a JVM agent. This is necessary because only a JVM agent can do all the things that it needs to do. Particularly, it needs to patch the running JDK itself to add the relevant checks to all the security-sensitive method calls.
why does Elasticsearch do this in 8.19.4 but not 8.18?
Because 8.18 only runs on JVMs that include the Java Security Manager, so we can use that for low-level protection against RCE exploits etc. 8.19 runs on JDK24 in which the Security Manager no longer exists so it requires the entitlements framework to achieve the same protections.
@DavidTurner I am trying to reproduce a simpler failure without elasticsearch in the equation to understand it better. Can you tell me more about the setup? Or maybe you can review what I’m doing and tell me how it differs from what elasticsearch is doing?
I have a java application #1 which just opens a file on the volume with the interesting permissions and then it just waits forever with the file open and never closes it (simulating the log files that elasticsearch might have open).
I have a java application #2 which does the following code:
VirtualMachine vm = VirtualMachine.attach(pid);
vm.loadAgent(agentPath, "someArgument");
vm.detach();
which is all the entitlement code does, really. I also have a simple agent I created, that’s specified by that agentPath
I execute java application #2 by feeding in an argument of the PID of java application #1.
However, I cannot get this to fail. It always successfully loads the agent and detaches without any problem.
I also tried a single application that self-attaches. When using my own agent the agent loads fine. When specifying elasticsearch-entitlement-agent-8.19.4.jar as the agent and the expected arg of org.elasticsearch.entitlement.initialization.EntitlementInitialization it fails presumably because only the Elasticsearch server has EntitlementInitialization loaded. I guess what I’m saying is that the problematic environment doesn’t have a problem attaching and loading agents by itself.
My next step will be to give you all a standard installation that fails / repros.
I have a java application #1 which just opens a file on the volume with the interesting permissions and then it just waits forever with the file open and never closes it (simulating the log files that elasticsearch might have open).
I’m not sure this is at all relevant.
I execute java application #2 by feeding in an argument of the PID of java application #1.
Elasticsearch is connecting back to itself, not to a separate application.
Apart from that, I don’t see anything particularly different from Elasticsearch based on this very high-level description of your actions. There will be a difference of course, it’s just that we can’t see it at this level.
Are you sure you’re running the applications in exactly the same fashion as Elasticsearch? Are you sure it’s successfully opening a socket at path /proc/230/root/tmp/.java_pid230?
Here’s a repro of the issue (finally - sorry it took so long). This is a k8s manifest that creates a read-write-many volume that elasticsearch will use as its logs volume. You can just run kubectl apply -f es-with-pvc.yaml (I’ve named the manifest below es-with-pvc.yaml). This should presumably work if you’re using aws, since that efs-sc storage class is for Amazon EFS. This has a default gid of 50000 but depending on the CSI drivers configs, the gid range used may be dynamic.
I anticipate that you will say that the volume needs to have the right permissions. And I’d agree with that. But what’s interesting is that even though we’re using a different user (elasticsearch user) we can still write files to the volume.
elasticsearch@elasticsearch-56f947754-kl4rr:~$ ls -al /usr/share/elasticsearch/logs
total 644
drwx------ 3 50017 50017 6144 Oct 22 13:57 .
drwxrwxr-x 1 root root 17 Sep 16 22:14 ..
-rw-rw-r-- 1 50017 50017 4126 Oct 22 13:57 gc.log
-rw-rw-r-- 1 50017 50017 62326 Oct 22 13:47 gc.log.00
-rw-rw-r-- 1 50017 50017 71722 Oct 22 13:48 gc.log.01
-rw-rw-r-- 1 50017 50017 63502 Oct 22 13:50 gc.log.02
-rw-rw-r-- 1 50017 50017 65675 Oct 22 13:51 gc.log.03
-rw-rw-r-- 1 50017 50017 54666 Oct 22 13:52 gc.log.04
-rw-rw-r-- 1 50017 50017 49971 Oct 22 13:54 gc.log.05
-rw-rw-r-- 1 50017 50017 87179 Oct 22 13:54 gc.log.06
-rw-rw-r-- 1 50017 50017 91678 Oct 22 13:56 gc.log.07
-rw-rw-r-- 1 50017 50017 69718 Oct 22 13:57 gc.log.08
drwx------ 22 50017 50017 6144 Oct 8 16:52 shared-logs
elasticsearch@elasticsearch-56f947754-kl4rr:~$ whoami
elasticsearch
elasticsearch@elasticsearch-56f947754-kl4rr:~$ id
uid=1000(elasticsearch) gid=1000(elasticsearch) groups=1000(elasticsearch),0(root),1(daemon)
elasticsearch@elasticsearch-56f947754-kl4rr:~$ touch /usr/share/elasticsearch/logs/asdf.txt
elasticsearch@elasticsearch-56f947754-kl4rr:~$ ls /usr/share/elasticsearch/logs/
asdf.txt gc.log gc.log.00 gc.log.01 gc.log.02 gc.log.03 gc.log.04 gc.log.05 gc.log.06 gc.log.07 gc.log.08 shared-logs
I’m honestly not sure what to think here. Clearly we can write to this log location but somehow if we make the logs volume owned by elasticsearch user, then the issue goes away. If it’s a permissions problem then we would expect the problem to manifest in a different way (logs not writeable / permission error) instead of saying that the JVM attach is not supported.
I’m not familiar enough with EKS to understand what you’re saying here. Can you share a pair of minimally-different configurations, one of which works and the other of which does not?
If by “minimal” you mean without use of k8s, I don’t think this will repro without k8s being in the mix. Because on a barebones machine with elasticsearch natively installed, if you try to specify a logs dir on a volume that doesn’t have correct privileges, then it’ll fail immediately with write privilege issues.
Would you agree with me that, at first glance, a process running as uid other than 50017 would NOT be able to write to (or read from) files in that directory?
That it can do so is an (arguably) nice k8s feature, which you utilized.
But if I tried to check if I can write via something like [ -w "$DIR" ] ... then I’d probably answer no - you can test this in your pods!
If I tried by actually writing a file, e.g. with a touch command, and my write succeeded, then I’d answer yes?
i.e. depending on how the check is made, you get a different answer. eg. I notice that -w uses the faccessat2 system call on my linux system. If you strace the system calls from 8.18 vs 8.19 around the “can I access path.logs”, I’m going to bet the system calls are different.
Yeah I tried to communicate that this dir is writable but shouldn’t be under conventional unix permissions. I think there is some CSI-driver provided or k8s provided magic at play here.
It is writable by the container's logged-in user, but the logged-in user UID=500 doesn't match the volume's files' user/group ownership (e.g., UID/GID=50000); nor does the mount path have Group or Other permissions set to allow the logged-in user to write to it; and there is no 50000 user/group in the operating system for the logged-in user to even be a part of; the supplemental groups specified by the output of id do not indicate a group that would give the user permission to write to this directory...and despite all this, the logged-in user can still write to the directory!
And yes if [ -w <dir> ]; then echo "writable"; fi outputs “writable” and also I can touch / write to a file in that dir.
What I tried to communicate is that this is, er, a bit niche. e.g. I rather doubt there is a test case in Elasticsearch for your specific setup, they might think to add one !?
Even though there is k8s / AWS / CSI-driver magic going on, I don’t really know where you are going with this specific thread? What are you expecting? Elastic aren’t going to roll-back the entitlements stuff, all you can realistically achieve, in short term, IMO, is a better understanding. And the (interesting) thread is 16 days young already!!
I dunno if you tried adding some supplementalGroups settings, might also be worth a try.
IF you trace all the syscalls you might get better clues, but obviously with k8s there’s multiple levels / a complex setup, so it’s not going to be easy.
I don’t really know where you are going with this specific thread? What are you expecting?
Knowing the exact root cause - which we have NOT yet found - could guide us on how to make a recommendation on how to fix the issue or work around it.
Initially I wasn’t sure if there was a problem with Elasticsearch or our setup. We have shed enough light on the situation to say there are definitely oddities with our setup and, for now, we can workaround the problem and there doesn’t seem to be any action on the Elasticsearch side.
But we have NOT really understood what the problem is with the JVM attach and how it relates to the log directory’s volume. Remember that the socket file it complains about is in /tmp which is NOT the directory on which the volume with the odd permissions is mounted.
We can vaguely say that there is some permission problem with log files that somehow relates to being able to attach to the JVM but that’s all very hand wavy.
This is a discussion forum thread, it’s not a commercial support case. Plus the effort most people would require to setup an env to reproduce your issue is simply too great, I’ve already tried but the “simplest” steps to reproduce would require me to spin up an k8s cluster, setup EFS, …. It’s, sorry, not gonna happen for the majority of forum contributors. As your setup is a bit niche, the issue you have hit is likely not impacting a lot of users …
You can try help yourself by tracing syscalls and trying to learn from that, keep trying to simplify things more, hope for a eureka contribution or for someone else to invest more time and respond - but you have no SLA here. Thats why I mentioned the age of the thread, 16 days, how long are you prepared to wait? Put another way, if this was blocking me, I’d also be looking for other ways forward too, so I hope you are doing that.
I said “minimally-different” not “minimal” - you need to narrow down the difference between a working and a broken configuration. Swapping out EFS for EBS is far too large a difference to be useful. You were earlier saying that you thought the permissions on a log directory were the important factor: if so, the difference between the working and broken configs should just be these permissions.
Maybe that helps you move forwards but if not then @RainTown is right, we probably need to break out strace and compare the actual sequence of syscalls between the working and broken versions to see where the difference in behaviour is coming from.
@RainTown is also right that I’m not going to go and set up all this AWS infrastructure to try and reproduce what you’re seeing. Elasticsearch itself doesn’t know anything about k8s or EBS-vs-EFS, it’s fundamentally just a plain old userspace application that interacts with the kernel through the regular syscall interface. Your kernel, in your specific configuration, is rejecting one of these syscalls.
My understanding, after much troubleshooting and experimenting on his part, is that the minimal difference between working and crashing on startup are the path.logs directory permissions/ownership, no difference in elasticsearch configuration, merely the version:
These directories are EFS mounts, which is the only known way to reproduce the issue, and it only works at all for the latter (50017) case, even in previous versions, due to the fsGroup options that k8s allows.
He’s now stuck. And as you noted:
None of us can yet see how the permissions/ownership of the log directory connect to the error he sees. Whether there’s an underlying elasticsearch bug or not, the error message that he sees is really unhelpful. Thus it’s an interesting puzzle, which is primarily why it interested me, but I’ve reached end of my road.
Often posters here don’t do much to help themselves. By contrast, I think @buitcj has made considerable efforts. We’re almost at “Help me, Obi-Wan. You’re my only hope.”
working on 8.18.x: drwx------ 3 50017 50017 6144 Oct 8 16:52 logs
TBC this is surprising to me independent of the entitlements thing. Elasticsearch needs to be able to write logs. I’m amazed it doesn’t fail in this situation.
Anyway if you run out of other ideas (and you can get it to pass/fail with just a logs path permissions change while using EFS in both cases) then the next step is strace:
we probably need to break out strace and compare the actual sequence of syscalls between the working and broken versions to see where the difference in behaviour is coming from.
This is all called from Elasticsearch.main pretty early in startup so there aren’t lots of interleaved threads or other stuff to worry about.
It can write the logs, via the fsGroup settings. It looks like it will fail, but actually it won’t. Even a [ -w <dir> ] test apparently returns true.
Magic, witchcraft, …
Though it’s not explicitly written, the same directory is presumably EFS/NFS “mounted” elsewhere, simultaneously to ES writing logs to it, I guess so some app can read the logs in close to real time, as @buitcj characterized this unusual setup is for “app specific reasons”.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
