Elasticsearch 7.9 doesn't start

Peter123 · August 22, 2020, 7:24pm

I have Elasticsearch 7.9 and it doesn't start. How fix this problem?

Centos 7
Java version
openjdk version "1.8.0_262"
OpenJDK Runtime Environment (build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)

#sudo systemctl start elasticsearch.service
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

journalctl -xe

    Starting Elasticsearch...
        -- Subject: Unit elasticsearch.service has begun start-up
        -- Defined-By: systemd
        -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
        --
        -- Unit elasticsearch.service has begun starting up.
    dovecot[2155]: imap-login: Login: user=<cpanel-ccs>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=8188, TLS, session=<tFy2AXytVK1/AAAB>
    dovecot[2155]: imap(cpanel-ccs)<8188><tFy2AXytVK1/AAAB>: Logged out in=50, out=862, bytes=50/862
    systemd-entrypoint[7971]: fatal error in thread [main], exiting
    systemd-entrypoint[7971]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
    systemd-entrypoint[7971]: at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34)
    systemd-entrypoint[7971]: at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
    systemd-entrypoint[7971]: at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33)
    systemd-entrypoint[7971]: at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:126)
    systemd-entrypoint[7971]: at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:137)
    systemd-entrypoint[7971]: at java.base/java.util.ArrayList.forEach(ArrayList.java:1510)
    systemd-entrypoint[7971]: at org.elasticsearch.node.Node.start(Node.java:868)
    systemd-entrypoint[7971]: at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:317)
    systemd-entrypoint[7971]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:402)
    systemd-entrypoint[7971]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
    systemd-entrypoint[7971]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
    systemd-entrypoint[7971]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
    systemd-entrypoint[7971]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
    systemd-entrypoint[7971]: at org.elasticsearch.cli.Command.main(Command.java:90)
    systemd-entrypoint[7971]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
    systemd-entrypoint[7971]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
    systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
    systemd[1]: Failed to start Elasticsearch.
        -- Subject: Unit elasticsearch.service has failed
        -- Defined-By: systemd
        -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
        --
        -- Unit elasticsearch.service has failed.
        --
        -- The result is failed.
    systemd[1]: Unit elasticsearch.service entered failed state.
    systemd[1]: elasticsearch.service failed.

var/log/elasticsearch/elasticsearch.log

[ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [main], exiting
        java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
                at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
                at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
                at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
                at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:126) ~[?:?]
                at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:137) ~[?:?]
                at java.util.ArrayList.forEach(ArrayList.java:1510) ~[?:?]
                at org.elasticsearch.node.Node.start(Node.java:868) ~[elasticsearch-7.9.0.jar:7.9.0]
                at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:317) ~[elasticsearch-7.9.0.jar:7.9.0]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:402) ~[elasticsearch-7.9.0.jar:7.9.0]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.0.jar:7.9.0]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.0.jar:7.9.0]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.0.jar:7.9.0]
                at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.0.jar:7.9.0]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.0.jar:7.9.0]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.0.jar:7.9.0]

#sudo systemctl status elasticsearch

elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2020-08-22 22:49:51 EEST; 8min ago
Docs: https://www.elastic.co
Process: 16498 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 16498 (code=exited, status=1/FAILURE)
CGroup: /system.slice/elasticsearch.service

systemd-entrypoint[16498]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
systemd-entrypoint[16498]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
systemd-entrypoint[16498]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
systemd-entrypoint[16498]: at org.elasticsearch.cli.Command.main(Command.java:90)
systemd-entrypoint[16498]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
systemd-entrypoint[16498]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start Elasticsearch.
systemd[1]: Unit elasticsearch.service entered failed state.
systemd[1]: elasticsearch.service failed.

Tested and doesn't work: https://stackoverflow.com/a/63422363

1. Set ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp" in /etc/sysconfig/elasticsearch
2. mkdir -p /var/lib/elasticsearch/tmp
3. chown -R elasticsearch.elasticsearch /var/lib/elasticsearch/tmp

Steve_Mushero · August 23, 2020, 1:36am

Welcome them to the community!

As that Stack Overflow mentions, it's usually due to /tmp mounted noexec and is similar to this Gihub issue:

Are you sure those Java options are being picked up? If so, they should show in the log as part of the error - the ES_JAVA_OPTS might be overridden or reset so always make sure the app is seeing the settings you think it is.

Also, are you mounting the default /tmp dir noexec and maybe try changing that, at least to test? Likely have to reboot for that.

Note that Stackoverflow also mentions RAM size, as they got this error with only 1GB of RAM in the VM; I hope you have more.

Also note that ES 7 uses its own bundled JVM, version 14, and won't use your version 8.

Peter123 · August 23, 2020, 10:42am

This mount mean this command mount -o remount,noexec /tmp ?

I need make something changes to file /etc/sysconfig/elasticsearch ?

################################
# Elasticsearch
################################

# Elasticsearch home directory
#ES_HOME=/usr/share/elasticsearch

# Elasticsearch Java path
#JAVA_HOME=

# Elasticsearch configuration directory
# Note: this setting will be shared with command-line tools
ES_PATH_CONF=/etc/elasticsearch

# Elasticsearch PID directory
#PID_DIR=/var/run/elasticsearch

# Additional Java OPTS
#ES_JAVA_OPTS=

# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true

################################
# Elasticsearch service
################################

Steve_Mushero · August 24, 2020, 1:07am

Well, this all depends on how /tmp is mounted now - you can see that in your /etc/fstab or just running mount - you may see defaults which normally includes exec, so if /tmp is mounted noexec you'll see it; just remove in fstab and reboot is easiest way.

Then use everything else normally.

This is IF this is the problem; if your /tmp is not separately mounted (common only in CentOS physical, some VM installs); if there is no /tmp mount (df -h will tell you, too) then your RAM may be too small, else I have no idea.

system · September 21, 2020, 1:07am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.