Elasticsearch 7.4 won't start properly

phucvandinh · October 2, 2019, 4:07am

I have a problem after upgrade elasticsearch from 7.3 to 7.4 on my dedicated server, it won't be able to start
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2019-10-02 07:47:50 +08; 4h 17min ago
Docs: http://www.elastic.co
Process: 3532 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 3532 (code=exited, status=1/FAILURE)
CGroup: /system.slice/elasticsearch.service

Same error happened with a test VPS install 7.4

Jugsofbeer · October 2, 2019, 4:43am

i tried creating a v7.4.0 deployment in ECE and it failed to start up which is unusual but sounds like a common issue.

DavidTurner · October 2, 2019, 6:29am

Can you share the Elasticsearch logs? The logs from systemd are not useful, but Elasticsearch will have shared much more information in its own log file.

phucvandinh · October 2, 2019, 8:12am

https://www24.zippyshare.com/v/0SODVTCx/file.html
Since i don't know which file is the correct one so I zip the whole folder so you can take a look.

DavidTurner · October 2, 2019, 8:20am

Thanks, too much information is always better than not enough. However I can't download anything from that link:

phucvandinh · October 2, 2019, 8:31am

When I click to the link, it goes correctly to the file @@
How about this one https://drive.google.com/uc?export=download&id=1gfqRgcpVK5XKtkerCilAXsfdG1SiF83i

DavidTurner · October 2, 2019, 8:47am

Got it, thanks. Here's the relevant bits:

[2019-10-02T16:06:32,066][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-6634371579246137060/jna--1985354563/jna258963392980261791.tmp: /tmp/elasticsearch-6634371579246137060/jna--1985354563/jna258963392980261791.tmp: failed to map seg
ment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
        at java.lang.System.load(System.java:1870) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:333) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.4.0.jar:7.4.0]
[2019-10-02T16:06:32,074][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot check if running as root because JNA is not available
[2019-10-02T16:06:32,074][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot install system call filter because JNA is not available
[2019-10-02T16:06:32,075][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot register console handler because JNA is not available
[2019-10-02T16:06:32,076][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2019-10-02T16:06:32,076][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot getrlimit RLIMIT_AS because JNA is not available
[2019-10-02T16:06:32,076][WARN ][o.e.b.Natives            ] [hydrogen.vietfriend.org] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2019-10-02T16:06:32,243][INFO ][o.e.e.NodeEnvironment    ] [hydrogen.vietfriend.org] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [42.7gb], net total_space [49.9gb], types [rootfs]
...
[2019-10-02T16:06:39,371][INFO ][o.e.n.Node               ] [hydrogen.vietfriend.org] started
[2019-10-02T16:06:39,374][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [hydrogen.vietfriend.org] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:66) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:74) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1507) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:768) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.0.jar:7.4.0]

This looks like it might be related to the systemd integration added in #44673. I will open an issue.

DavidTurner · October 2, 2019, 8:52am

I think this manual page matches the problem you are facing, and gives a remedy:

https://www.elastic.co/guide/en/elasticsearch/reference/current/executable-jna-tmpdir.html

Could you try that?

phucvandinh · October 2, 2019, 9:58am

I don't know if these steps are correct:
I add the below line to /etc/sysconfig/elasticsearch

ES_TMPDIR=/usr/share/elasticsarch/tmp

and add this

Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

after running this command systemctl edit --full elasticsearch.service

Still can't start elasticsearch...

DavidTurner · October 2, 2019, 10:00am

No, that's not what the instructions say.

... you must remount the temporary directory used for JNA to not be mounted with noexec .

The temporary directory in question is shown in the first log line:

java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-6634371579246137060/jna--1985354563/jna258963392980261791.tmp: /tmp/elasticsearch-6634371579246137060/jna--1985354563/jna258963392980261791.tmp: failed to map seg ment from shared object: Operation not permitted

You must make sure that this is not mounted noexec. An alternative is also suggested in the docs:

Alternatively, this location can be controlled with the JVM flag -Djna.tmpdir=<path>.

phucvandinh · October 2, 2019, 10:24am

I’m not sure what to do. I install Elasticsearch on Centos 7 via rpm packages. Everything works perfectly until I upgraded to 7.4 via yum cmd. I tried to locate that tmp folder via sftp but can’t find it @@

DavidTurner · October 2, 2019, 10:33am

I am not following. Are you saying you cannot find /tmp? Why are you using sftp? Is /tmp mounted noexec? When you tried setting ES_TMPDIR=/usr/share/elasticsarch/tmp (spelled wrong?) did the messages change? If so, what did they look like? Did you try setting -Djna.tmpdir=<path>?

Sophie1892 · October 14, 2019, 9:53am

I've removed it all.
And make a clean install.

By default it was working. And now I'm doing it step by step.

1.When I set network.host to elastic1.domain.xy it won't start again Official Website.
I get a message about non-loopback address .
And yes, before I get it here. I've read the Boostrap Checks but still it has only an explanation and not a guide to fix this.

DavidTurner · October 14, 2019, 10:14am

Hi @Sophie1892, this thread doesn't seem related to the problem you're asking about. Would you open a new topic instead? Also the message you are seeing will say a lot more than simply non-loopback address. It's not possible to help from just those few words. If you need help understanding the whole message then please share the whole message.

whitedogbe · October 16, 2019, 9:48pm

Hello, I installed elasticsearch for the first time ever today and am facing the exact same issue.
I installed version 7.4 on CentOS 7. I also have cPanel installed.

I didn't install Java as that was already present, it's used by solr (mailbox search, installed by cPanel).

Actually elasticsearch seems to start but then terminates after about 30 seconds.
It also seems to run if (as root) I execute this:
/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid

But not via systemctl:
sudo systemctl start elasticsearch.service

Unregistered Authentication Agent for unix-process:14733:16820115 (system bus name :1.12908, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)

system · November 13, 2019, 9:57pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
My Elasticsearch wont start after java update Elasticsearch	5	322	October 27, 2022
ElasticSearch StartUp Error, After updating from v 2.4.3 to v 2.4.4 Elasticsearch	5	971	March 22, 2017
Elasticsearch installation issues Elasticsearch	3	363	December 6, 2023
Failed to start Elasticsearch.7.15.2 Elasticsearch	7	2550	December 15, 2021
Elasticsearch fails to start on ubuntu 20.04 Elasticsearch	7	3584	February 27, 2021

Elasticsearch 7.4 won't start properly

Related topics