Elasticsearch no longer works under systemd (7.4.0 on CentOS 7.7.1908)?

[IanC (Ian Chesal)]

I'm at a loss here. I have a WHM-managed CentOS 7.7 that's regularly applying patches in the background. I did a reboot today to have a kernel update take effect and when the system came back up ElasticSearch failed to start.

I'm running the latest ES package that's available according to yum on my machine:

# yum info elasticsearch
Loaded plugins: langpacks, universal-hooks
Installed Packages
Name        : elasticsearch
Arch        : x86_64
Version     : 7.4.0
Release     : 1
Size        : 469 M
Repo        : installed
From repo   : elasticsearch-7.x
Summary     : Distributed RESTful search engine built for the cloud
URL         : https://www.elastic.co/
License     : Elastic License
Description : Reference documentation can be found at
            : https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
            : and the 'Elasticsearch: The Definitive Guide' book can be found at
            : https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html

And my OS is:

# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)

The problem seems to be releated to ES_SD_NOTIFY=true -- if this is set to true I get:

[2019-10-01T12:59:52,782][INFO ][o.e.n.Node               ] [server] started
[2019-10-01T12:59:52,784][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [sever] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:66) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:74) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1507) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:768) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.0.jar:7.4.0]

When the ES process is starting up. If I set it to false ES starts, but then systemd is never notified that ES has started correctly and, as a result, times out the startup thinking ES failed to start and shuts down ES.

I'm sort of at a loss for how to resolve this one. Anyone have any pointers? I'm using the rpm-installed ES so it has its own JDK.

[IanC (Ian Chesal)]

Figured it out. /tmp is set to mount with noexec on the system and WHM seems to want to enforce that pretty hard. I just set ES_TMPDIR to point to another location and now everything seems to work correctly.

Leaving this here as the answer for anyone else that runs into the problem.

[IanC (Ian Chesal)]

I'll add that I set ES_TMPDIR in two places. First I set it in /etc/sysconfig/elasticsearch and then I ran systemctl edit --full elasticsearch.service and added:

Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

In the [service] section. I'm not 100% certain the second step is required but systemd is such an utter mess I can't be certain it was or was not.