Just thought I'd post an update. These errors have been cropping up over and over. Always on the same node.
Yesterday I had it all working, upgraded to 7.9.1 (successfully as far as I could tell), and then a bit later ES crashed on the node. I eventually, I thought, got it running again by restarting a whole bunch of times.
Then today the node wasn't responding, even though ES was running. So I restarted ES. Which failed with the "Service lookup key" error. And it kept failing. Sometimes with that error, other times with the "whitelist" error, and even saw some like An exception was caught and reported. Message: access denied ("java.lang.RuntimePermission" "accessClassInPackage.jdk.internal.vm.annotation") at _unknown_
.
Searching on the first two errors led me back to this topic... The permission error just led me to posts about people having the wrong java version. Which isn't my issue. I only have whatever java ES comes with on my node. (I did double check that.)
Eventually, after running systemctl restart elasticsearch
a bunch of times, the node started up successfully.
I am convinced that there is something odd on my node that causes this. Anyone have any suggestions on what to look for?