Elasticsearch service will not start on Ubuntu server 18.04

miceffre · February 16, 2020, 8:32pm

I get the following error: "Job for elasticsearch.service failed because the control process exited with error code." When I try to start elasticsearch on my Ubuntu 18.04 server. I have tried uninstalling and re-installing elasticsearch several times and I always get the same resulting error.

Some system info:
Operating System: Ubuntu 18.04.4 LTS
Kernel: Linux 4.15.0-76-generic
Architecture: x86-64

Java
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode)

elasticsearch/stable,now 7.6.0 amd64 [installed]

systemctl status elasticsearch.service

elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: failed (Result: exit-code) since Sun 2020-02-16 20:21:01 UTC; 8min ago
Docs: http://www.elastic.co
Process: 7533 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid -
Main PID: 7533 (code=exited, status=1/FAILURE)

Feb 16 20:21:00 archie elasticsearch[7533]: at org.elasticsearch.bootstrap.Elasticsearch.execu
Feb 16 20:21:00 archie elasticsearch[7533]: at org.elasticsearch.cli.EnvironmentAwareCommand.e
Feb 16 20:21:00 archie elasticsearch[7533]: at org.elasticsearch.cli.Command.mainWithoutErrorH
Feb 16 20:21:00 archie elasticsearch[7533]: at org.elasticsearch.cli.Command.main(Command.java
Feb 16 20:21:00 archie elasticsearch[7533]: at org.elasticsearch.bootstrap.Elasticsearch.main(
Feb 16 20:21:00 archie elasticsearch[7533]: at org.elasticsearch.bootstrap.Elasticsearch.main(
Feb 16 20:21:00 archie elasticsearch[7533]: For complete error details, refer to the log at /var/log/e
Feb 16 20:21:01 archie systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/F
Feb 16 20:21:01 archie systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Feb 16 20:21:01 archie systemd[1]: Failed to start Elasticsearch.

edit /var/log/elasticsearch/gc.log.10

[2020-02-16T20:20:34.745+0000][7604][gc] Using Concurrent Mark Sweep
[2020-02-16T20:20:34.745+0000][7604][gc,heap,coops] Heap address: 0x00000000c0000000, size: 1024 MB, Compressed Oops mode: 32-bit
[2020-02-16T20:20:34.851+0000][7604][safepoint ] Safepoint "EnableBiasedLocking", Time since last: 53024080 ns, Reaching safepoint: 105157 ns, At safepoint: 62857 ns, Total: 168014 ns
[2020-02-16T20:20:34.857+0000][7604][gc,heap,exit ] Heap
[2020-02-16T20:20:34.857+0000][7604][gc,heap,exit ] par new generation total 153344K, used 8179K [0x00000000c0000000, 0x00000000ca660000, 0x00000000ca660000)
[2020-02-16T20:20:34.857+0000][7604][gc,heap,exit ] eden space 136320K, 6% used [0x00000000c0000000, 0x00000000c07fce60, 0x00000000c8520000)
[2020-02-16T20:20:34.857+0000][7604][gc,heap,exit ] from space 17024K, 0% used [0x00000000c8520000, 0x00000000c8520000, 0x00000000c95c0000)
[2020-02-16T20:20:34.857+0000][7604][gc,heap,exit ] to space 17024K, 0% used [0x00000000c95c0000, 0x00000000c95c0000, 0x00000000ca660000)
[2020-02-16T20:20:34.857+0000][7604][gc,heap,exit ] concurrent mark-sweep generation total 878208K, used 0K [0x00000000ca660000, 0x0000000100000000, 0x0000000100000000)
[2020-02-16T20:20:34.857+0000][7604][gc,heap,exit ] Metaspace used 136K, capacity 4480K, committed 4480K, reserved 1056768K
[2020-02-16T20:20:34.857+0000][7604][gc,heap,exit ] class space used 3K, capacity 384K, committed 384K, reserved 1048576K

spinscale · February 17, 2020, 10:42am

Hey,

can you share more output where you saw that exception stack trace, the last few lines are not enough. You can use journalctl to do that, see https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#rpm-running-systemd

My assumption here, is that a bootstrap check fails, however if that happens, there is usually a detailed explanation or a link in the error message itself, so taking a closer look at the logs usually helps.

--Alex

miceffre · February 17, 2020, 1:38pm

Sure, I will work on getting you that right now. I have seen some references to the bootstrap in the /var/.../escluster01.log. I will work on getting you the full dump but for now here is the last timestamped error from the dump.
[2020-02-16T17:14:38,075][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-archie] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].]

miceffre · February 17, 2020, 1:41pm

oh... I have not tried the logs recommended solution yet either. [xpack.ml.enabled:false]

systemctl status elasticsearch.service

gist.github.com

https://gist.github.com/miceffre/5a5b9a608c9a5f95898c4041e442ccca

systemctl status

 elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Mon 2020-02-17 10:14:38 EST; 21min ago
     Docs: http://www.elastic.co
  Process: 6070 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid (code=exited, status=1/FAILURE)
 Main PID: 6070 (code=exited, status=1/FAILURE)

Feb 17 10:14:38 archie elasticsearch[6070]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)

This file has been truncated. show original

journalctl --since "1 hour ago"

gist.github.com

https://gist.github.com/miceffre/7b173bb83d1024cb2713e02f6e31b032

gistfile1.txt

Feb 17 10:14:10 archie systemd[1]: Starting Elasticsearch...
Feb 17 10:14:11 archie elasticsearch[6070]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Feb 17 10:14:15 archie elasticsearch[6070]: [2020-02-17T10:14:15,178][INFO ][o.e.e.NodeEnvironment    ] [es-archie] using [1] data paths, mounts [[/var (/dev/sda3)]], net usable_space [161.6gb], net total_space
Feb 17 10:14:15 archie elasticsearch[6070]: [2020-02-17T10:14:15,184][INFO ][o.e.e.NodeEnvironment    ] [es-archie] heap size [1007.3mb], compressed ordinary object pointers [true]
Feb 17 10:14:15 archie elasticsearch[6070]: [2020-02-17T10:14:15,255][INFO ][o.e.n.Node               ] [es-archie] node name [es-archie], node ID [fjZ0LUQXS-e_dVG7p4m9mA], cluster name [escluster01]
Feb 17 10:14:15 archie elasticsearch[6070]: [2020-02-17T10:14:15,257][INFO ][o.e.n.Node               ] [es-archie] version[7.6.0], pid[6070], build[default/deb/7f634e9f44834fbc12724506cc1da681b0c3b1e3/2020-02-0
Feb 17 10:14:15 archie elasticsearch[6070]: [2020-02-17T10:14:15,259][INFO ][o.e.n.Node               ] [es-archie] JVM home [/usr/share/elasticsearch/jdk]
Feb 17 10:14:15 archie elasticsearch[6070]: [2020-02-17T10:14:15,260][INFO ][o.e.n.Node               ] [es-archie] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX
Feb 17 10:14:18 archie elasticsearch[6070]: [2020-02-17T10:14:18,794][INFO ][o.e.p.PluginsService     ] [es-archie] loaded module [aggs-matrix-stats]
Feb 17 10:14:18 archie elasticsearch[6070]: [2020-02-17T10:14:18,796][INFO ][o.e.p.PluginsService     ] [es-archie] loaded module [analysis-common]

This file has been truncated. show original

miceffre · February 17, 2020, 9:02pm

I was able to fix the configuration to enable the elasticsearch server to start by updating the elasticsearch.yml file with the following:

------------------------------ Machine Learning ------------------------------

xpack.ml.enabled: false

The computer I'm using is an older PC and this disables the machine learning feature that is enabled by default.

system · March 16, 2020, 9:02pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.