Elasticsearch 7.4 won't start properly

I have a problem after upgrade elasticsearch from 7.3 to 7.4 on my dedicated server, it won't be able to start
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2019-10-02 07:47:50 +08; 4h 17min ago
Docs: http://www.elastic.co
Process: 3532 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 3532 (code=exited, status=1/FAILURE)
CGroup: /system.slice/elasticsearch.service

Same error happened with a test VPS install 7.4

i tried creating a v7.4.0 deployment in ECE and it failed to start up which is unusual but sounds like a common issue.

Can you share the Elasticsearch logs? The logs from systemd are not useful, but Elasticsearch will have shared much more information in its own log file.

https://www24.zippyshare.com/v/0SODVTCx/file.html
Since i don't know which file is the correct one so I zip the whole folder so you can take a look.

Thanks, too much information is always better than not enough. However I can't download anything from that link:

When I click to the link, it goes correctly to the file @@
How about this one https://drive.google.com/uc?export=download&id=1gfqRgcpVK5XKtkerCilAXsfdG1SiF83i

Got it, thanks. Here's the relevant bits:

[2019-10-02T16:06:32,066][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-6634371579246137060/jna--1985354563/jna258963392980261791.tmp: /tmp/elasticsearch-6634371579246137060/jna--1985354563/jna258963392980261791.tmp: failed to map seg
ment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
        at java.lang.System.load(System.java:1870) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:333) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.4.0.jar:7.4.0]
[2019-10-02T16:06:32,074][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot check if running as root because JNA is not available
[2019-10-02T16:06:32,074][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot install system call filter because JNA is not available
[2019-10-02T16:06:32,075][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot register console handler because JNA is not available
[2019-10-02T16:06:32,076][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2019-10-02T16:06:32,076][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot getrlimit RLIMIT_AS because JNA is not available
[2019-10-02T16:06:32,076][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2019-10-02T16:06:32,243][INFO ][o.e.e.NodeEnvironment    ] [hydrogen.vietfriend.org] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [42.7gb], net total_space [49.9gb], types [rootfs]
...
[2019-10-02T16:06:39,371][INFO ][o.e.n.Node               ] [hydrogen.vietfriend.org] started
[2019-10-02T16:06:39,374][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [hydrogen.vietfriend.org] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:66) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:74) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1507) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:768) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.0.jar:7.4.0]

This looks like it might be related to the systemd integration added in #44673. I will open an issue.

I think this manual page matches the problem you are facing, and gives a remedy:

https://www.elastic.co/guide/en/elasticsearch/reference/current/executable-jna-tmpdir.html

Could you try that?

I don't know if these steps are correct:
I add the below line to /etc/sysconfig/elasticsearch

ES_TMPDIR=/usr/share/elasticsarch/tmp

and add this

Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

after running this command systemctl edit --full elasticsearch.service

Still can't start elasticsearch...

No, that's not what the instructions say.

... you must remount the temporary directory used for JNA to not be mounted with noexec .

The temporary directory in question is shown in the first log line:

java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-6634371579246137060/jna--1985354563/jna258963392980261791.tmp: /tmp/elasticsearch-6634371579246137060/jna--1985354563/jna258963392980261791.tmp: failed to map seg ment from shared object: Operation not permitted

You must make sure that this is not mounted noexec. An alternative is also suggested in the docs:

Alternatively, this location can be controlled with the JVM flag -Djna.tmpdir=<path>.

1 Like

I’m not sure what to do. I install Elasticsearch on Centos 7 via rpm packages. Everything works perfectly until I upgraded to 7.4 via yum cmd. I tried to locate that tmp folder via sftp but can’t find it @@

I am not following. Are you saying you cannot find /tmp? Why are you using sftp? Is /tmp mounted noexec? When you tried setting ES_TMPDIR=/usr/share/elasticsarch/tmp (spelled wrong?) did the messages change? If so, what did they look like? Did you try setting -Djna.tmpdir=<path>?

I've removed it all.
And make a clean install.

By default it was working. And now I'm doing it step by step.

1.When I set network.host to elastic1.domain.xy it won't start again Official Website.
I get a message about non-loopback address .
And yes, before I get it here. I've read the Boostrap Checks but still it has only an explanation and not a guide to fix this.

Hi @Sophie1892, this thread doesn't seem related to the problem you're asking about. Would you open a new topic instead? Also the message you are seeing will say a lot more than simply non-loopback address. It's not possible to help from just those few words. If you need help understanding the whole message then please share the whole message.

Hello, I installed elasticsearch for the first time ever today and am facing the exact same issue.
I installed version 7.4 on CentOS 7. I also have cPanel installed.

I didn't install Java as that was already present, it's used by solr (mailbox search, installed by cPanel).

Actually elasticsearch seems to start but then terminates after about 30 seconds.
It also seems to run if (as root) I execute this:
/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid

But not via systemctl:
sudo systemctl start elasticsearch.service

Unregistered Authentication Agent for unix-process:14733:16820115 (system bus name :1.12908, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.