I'm at a loss here. I have a WHM-managed CentOS 7.7 that's regularly applying patches in the background. I did a reboot today to have a kernel update take effect and when the system came back up ElasticSearch failed to start.
I'm running the latest ES package that's available according to yum on my machine:
# yum info elasticsearch
Loaded plugins: langpacks, universal-hooks
Installed Packages
Name : elasticsearch
Arch : x86_64
Version : 7.4.0
Release : 1
Size : 469 M
Repo : installed
From repo : elasticsearch-7.x
Summary : Distributed RESTful search engine built for the cloud
URL : https://www.elastic.co/
License : Elastic License
Description : Reference documentation can be found at
: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
: and the 'Elasticsearch: The Definitive Guide' book can be found at
: https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
And my OS is:
# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
The problem seems to be releated to ES_SD_NOTIFY=true
-- if this is set to true
I get:
[2019-10-01T12:59:52,782][INFO ][o.e.n.Node ] [server] started
[2019-10-01T12:59:52,784][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [sever] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:66) ~[?:?]
at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:74) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1507) ~[?:?]
at org.elasticsearch.node.Node.start(Node.java:768) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.0.jar:7.4.0]
When the ES process is starting up. If I set it to false ES starts, but then systemd is never notified that ES has started correctly and, as a result, times out the startup thinking ES failed to start and shuts down ES.
I'm sort of at a loss for how to resolve this one. Anyone have any pointers? I'm using the rpm-installed ES so it has its own JDK.