Ailed to start Elasticsearch. (code=killed, signal=KILL)

Tealk · July 15, 2021, 5:46pm

hello i just want to move my mastodon server and i just installed Elasticsearch on a new server but i get the following error when i start it:

systemctl start elasticsearch
Job for elasticsearch.service failed because a fatal signal was delivered to the control process.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

systemctl status elasticsearch.service

* elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Thu 2021-07-15 17:02:36 UTC; 6min ago
     Docs: https://www.elastic.co
  Process: 51372 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
 Main PID: 51372 (code=killed, signal=KILL)
      CPU: 35.335s

Jul 15 17:02:13 Fediverse systemd[1]: Starting Elasticsearch...
Jul 15 17:02:36 Fediverse systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jul 15 17:02:36 Fediverse systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jul 15 17:02:36 Fediverse systemd[1]: Failed to start Elasticsearch.
Jul 15 17:02:36 Fediverse systemd[1]: elasticsearch.service: Consumed 35.335s CPU time.

journalctl -xe

Jul 15 17:45:24 Fediverse systemd[1]: Starting Elasticsearch...
-- Subject: A start job for unit elasticsearch.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit elasticsearch.service has begun execution.
-- 
-- The job identifier is 2933.
Jul 15 17:45:34 Fediverse systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- An ExecStart= process belonging to unit elasticsearch.service has exited.
-- 
-- The process' exit code is 'killed' and its exit status is 9.
Jul 15 17:45:34 Fediverse systemd[1]: elasticsearch.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit elasticsearch.service has entered the 'failed' state with result 'signal'.
Jul 15 17:45:34 Fediverse systemd[1]: Failed to start Elasticsearch.
-- Subject: A start job for unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit elasticsearch.service has finished with a failure.
-- 
-- The job identifier is 2933 and the job result is failed.
Jul 15 17:45:34 Fediverse systemd[1]: elasticsearch.service: Consumed 36.878s CPU time.
-- Subject: Resources consumed by unit runtime
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit elasticsearch.service completed and consumed the indicated resources.

/var/log/elasticsearch/gc.log

[2021-07-15T17:09:34.638+0000][51734][gc] Using G1
[2021-07-15T17:09:34.664+0000][51734][gc,init] Version: 16+36 (release)
[2021-07-15T17:09:34.664+0000][51734][gc,init] CPUs: 12 total, 6 available
[2021-07-15T17:09:34.664+0000][51734][gc,init] Memory: 64305M
[2021-07-15T17:09:34.664+0000][51734][gc,init] Large Page Support: Disabled
[2021-07-15T17:09:34.664+0000][51734][gc,init] NUMA Support: Disabled
[2021-07-15T17:09:34.664+0000][51734][gc,init] Compressed Oops: Enabled (Non-zero disjoint base)
[2021-07-15T17:09:34.664+0000][51734][gc,init] Heap Region Size: 16M
[2021-07-15T17:09:34.664+0000][51734][gc,init] Heap Min Capacity: 31G
[2021-07-15T17:09:34.664+0000][51734][gc,init] Heap Initial Capacity: 31G
[2021-07-15T17:09:34.664+0000][51734][gc,init] Heap Max Capacity: 31G
[2021-07-15T17:09:34.664+0000][51734][gc,init] Pre-touch: Disabled
[2021-07-15T17:09:34.664+0000][51734][gc,init] Parallel Workers: 6
[2021-07-15T17:09:34.664+0000][51734][gc,init] Concurrent Workers: 2
[2021-07-15T17:09:34.664+0000][51734][gc,init] Concurrent Refinement Workers: 6
[2021-07-15T17:09:34.664+0000][51734][gc,init] Periodic GC: Disabled
[2021-07-15T17:09:34.664+0000][51734][gc,metaspace] CDS disabled.
[2021-07-15T17:09:34.664+0000][51734][gc,metaspace] Compressed class space mapped at: 0x00000017c1000000-0x0000001801000000, reserved size: 1073741824
[2021-07-15T17:09:34.664+0000][51734][gc,metaspace] Narrow klass base: 0x00000017c1000000, Narrow klass shift: 0, Narrow klass range: 0x40000000
[2021-07-15T17:09:34.705+0000][51734][gc,heap,exit] Heap
[2021-07-15T17:09:34.705+0000][51734][gc,heap,exit]  garbage-first heap   total 32505856K, used 24578K [0x0000001001000000, 0x00000017c1000000)
[2021-07-15T17:09:34.705+0000][51734][gc,heap,exit]   region size 16384K, 2 young (32768K), 0 survivors (0K)
[2021-07-15T17:09:34.705+0000][51734][gc,heap,exit]  Metaspace       used 3562K, committed 3648K, reserved 1056768K
[2021-07-15T17:09:34.705+0000][51734][gc,heap,exit]   class space    used 282K, committed 320K, reserved 1048576K

spinscale · July 16, 2021, 8:45am

Can you share the log file output named like your cluster name from /var/log/elasticsearch? Thanks!

Tealk · July 16, 2021, 9:00am

I only have the following files in the folder

/var/log/elasticsearch# ls
gc.log  gc.log.00  gc.log.01  gc.log.02  gc.log.03  gc.log.04  gc.log.05  gc.log.06  gc.log.07  gc.log.08

gc.log

[2021-07-15T20:57:19.719+0000][111][gc] Using G1

gc.log.00

[2021-07-15T17:00:18.963+0000][51322][gc] Using G1
[2021-07-15T17:00:18.986+0000][51322][gc,init] Version: 16+36 (release)
[2021-07-15T17:00:18.986+0000][51322][gc,init] CPUs: 12 total, 6 available
[2021-07-15T17:00:18.986+0000][51322][gc,init] Memory: 64305M
[2021-07-15T17:00:18.986+0000][51322][gc,init] Large Page Support: Disabled
[2021-07-15T17:00:18.986+0000][51322][gc,init] NUMA Support: Disabled
[2021-07-15T17:00:18.986+0000][51322][gc,init] Compressed Oops: Enabled (Non-zero disjoint base)
[2021-07-15T17:00:18.986+0000][51322][gc,init] Heap Region Size: 16M
[2021-07-15T17:00:18.986+0000][51322][gc,init] Heap Min Capacity: 31G
[2021-07-15T17:00:18.986+0000][51322][gc,init] Heap Initial Capacity: 31G
[2021-07-15T17:00:18.986+0000][51322][gc,init] Heap Max Capacity: 31G
[2021-07-15T17:00:18.986+0000][51322][gc,init] Pre-touch: Disabled
[2021-07-15T17:00:18.986+0000][51322][gc,init] Parallel Workers: 6
[2021-07-15T17:00:18.986+0000][51322][gc,init] Concurrent Workers: 2
[2021-07-15T17:00:18.986+0000][51322][gc,init] Concurrent Refinement Workers: 6
[2021-07-15T17:00:18.986+0000][51322][gc,init] Periodic GC: Disabled
[2021-07-15T17:00:18.986+0000][51322][gc,metaspace] CDS disabled.
[2021-07-15T17:00:18.986+0000][51322][gc,metaspace] Compressed class space mapped at: 0x00000017c1000000-0x0000001801000000, reserved size: 1073741824
[2021-07-15T17:00:18.986+0000][51322][gc,metaspace] Narrow klass base: 0x00000017c1000000, Narrow klass shift: 0, Narrow klass range: 0x40000000
[2021-07-15T17:00:19.025+0000][51322][gc,heap,exit] Heap
[2021-07-15T17:00:19.025+0000][51322][gc,heap,exit]  garbage-first heap   total 32505856K, used 24578K [0x0000001001000000, 0x00000017c1000000)
[2021-07-15T17:00:19.025+0000][51322][gc,heap,exit]   region size 16384K, 2 young (32768K), 0 survivors (0K)
[2021-07-15T17:00:19.025+0000][51322][gc,heap,exit]  Metaspace       used 3561K, committed 3648K, reserved 1056768K
[2021-07-15T17:00:19.025+0000][51322][gc,heap,exit]   class space    used 282K, committed 320K, reserved 1048576K

this repeats

Tealk · July 16, 2021, 10:38pm

I don't know where from, but elasticsearch probably gets the size of the memory of the host server out, that's why the log also says the container has 64GB Ram. But the container has only 16 allocated and because the minimum size was calculated to 31GB the process could not start.
The following setting then fixed that:
Code:

etc/elasticsearch/jvm.options.d/jvm.options
-Xms4g
-Xmx4g

system · August 13, 2021, 10:39pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.