Elasticsearch Down

hi community
My elasticsearch Server is Down, ¿what could be happening?

Elasticsearch 8.7

1) systemctl status elasticsearch

elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: oom-kill) since Mon 2023-04-17 20:14:58 -05; 13min ago
Docs: https://www.elastic.co
Process: 72295 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143)
Main PID: 72295 (code=exited, status=143)
CPU: 19min 45.197s

Starting Elasticsearch...
Started Elasticsearch.
elasticsearch.service: A process of this unit has been killed by the OOM killer.
ERROR: Elasticsearch exited unexpectedly
elasticsearch.service: Failed with result 'oom-kill'.
elasticsearch.service: Consumed 19min 45.197s CPU time.

2) Logs /var/log/elasticsearch elasticsearch.log

root@XXX:/var/log/elasticsearch# tail -f elasticsearch.log
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
... 16 more
[2023-04-17T20:14:45,952][WARN ][o.e.h.n.Netty4HttpServerTransport] [XXX] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.0.1.20:9200, remoteAddress=/192.0.1.20:59818}
[2023-04-17T20:14:47,674][WARN ][o.e.h.n.Netty4HttpServerTransport] [XXX] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.0.1.20:9200, remoteAddress=/192.0.1.20:59822}
[2023-04-17T20:14:48,398][WARN ][o.e.h.n.Netty4HttpServerTransport] [XXX] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.0.1.20:9200, remoteAddress=/192.0.1.20:59832}
[2023-04-17T20:14:50,570][WARN ][o.e.h.n.Netty4HttpServerTransport] [XXX] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.0.1.20:9200, remoteAddress=/192.0.1.20:59840}

  1. journalctl -xe

abr 17 20:39:28 XXX kernel: [ 73840] 1000 73840 3682 308 69632 1 0 sshd
abr 17 20:39:28 XXX kernel: [ 73841] 1000 73841 2042 359 49152 28 0 bash
abr 17 20:39:28 XXX kernel: [ 73847] 1000 73847 2502 146 65536 4 0 su
abr 17 20:39:28 XXX kernel: [ 73848] 0 73848 1831 150 49152 3 0 bash
abr 17 20:39:28 XXX kernel: [ 73849] 0 73849 32706 164 184320 6 0 journalctl
abr 17 20:39:28 XXX kernel: [ 73850] 0 73850 1461 56 57344 4 0 less
abr 17 20:39:28 XXX kernel: [ 73991] 0 73991 3630 307 73728 0 0 sshd
abr 17 20:39:28 XXX kernel: [ 73997] 1000 73997 3682 309 73728 0 0 sshd
abr 17 20:39:28 XXX kernel: [ 73998] 1000 73998 2042 386 57344 0 0 bash
abr 17 20:39:28 XXX kernel: [ 74001] 1000 74001 2502 149 57344 0 0 su
abr 17 20:39:28 XXX kernel: [ 74002] 0 74002 1819 149 53248 0 0 bash
abr 17 20:39:28 XXX kernel: [ 74025] 0 74025 2535 68 57344 0 0 systemctl
abr 17 20:39:28 XXX kernel: [ 74026] 0 74026 3294 149 65536 0 0 systemd-tty-ask
abr 17 20:39:28 XXX kernel: [ 74027] 107 74027 650623 591 430080 17051 0 java
abr 17 20:39:28 XXX kernel: [ 74086] 107 74086 1170357 470781 4648960 72371 0 java
abr 17 20:39:28 XXX kernel: [ 74112] 107 74112 22876 130 77824 0 0 controller
abr 17 20:39:28 XXX kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/elasticsearch.service,task=java,pid=7408>
abr 17 20:39:28 XXX kernel: Out of memory: Killed process 74086 (java) total-vm:4681428kB, anon-rss:1883124kB, file-rss:0kB, shmem-rss:0kB, UID:107 pgtables:4540kB oom_score_adj:0
abr 17 20:39:28 XXX kibana[72442]: [2023-04-17T20:39:26.012-05:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: co>
abr 17 20:39:28 XXX kibana[72442]: [2023-04-17T20:39:26.642-05:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: co>
abr 17 20:39:30 XXX systemd-entrypoint[74027]: ERROR: Elasticsearch exited unexpectedly
abr 17 20:39:30 XXX systemd[1]: elasticsearch.service: Main process exited, code=exited, status=137/n/a
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ An ExecStart= process belonging to unit elasticsearch.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 137.
abr 17 20:39:30 XXX systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit elasticsearch.service has entered the 'failed' state with result 'exit-code'.
abr 17 20:39:30 XXX systemd[1]: Failed to start Elasticsearch.
░░ Subject: A start job for unit elasticsearch.service has failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ A start job for unit elasticsearch.service has finished with a failure.
░░
░░ The job identifier is 11055 and the job result is failed.
abr 17 20:39:30 XXX systemd[1]: elasticsearch.service: Consumed 55.067s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit elasticsearch.service completed and consumed the indicated resources.

4) Please help me ...

Looks like It ran out of memory / RAM and died.

What size server RAM is it on and what else is running on the server?

Did you manually set the JVM heap size or just used the defaults.

My test computer has 2CPU and 4GB of RAM, It only runs ELasticSearch and Kibana. it has the default JVM heap size setting. It is a new configuration.

Free -h
total used free shared buff/cache available
Mem: 3,8Gi 1,9Gi 1,8Gi 0,0Ki 120Mi 1,8Gi
Swap: 0B 0B 0B

systemctl status elasticsearch

● elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2023-04-17 22:18:54 -05; 1min 37s ago
Docs: https://www.elastic.co
Process: 75680 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=137)
Main PID: 75680 (code=exited, status=137)
CPU: 7.997s

abr 17 22:18:45 XXX systemd[1]: Starting Elasticsearch...
abr 17 22:18:53 XXX systemd-entrypoint[75680]: ERROR: Elasticsearch exited unexpectedly
abr 17 22:18:54 XXX systemd[1]: elasticsearch.service: Main process exited, code=exited, status=137/n/a
abr 17 22:18:54 XXX systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
abr 17 22:18:54 XXX systemd[1]: Failed to start Elasticsearch.
abr 17 22:18:54 XXX systemd[1]: elasticsearch.service: Consumed 7.997s CPU time.

Is there an easy way to view the logs?

I am checking in /var/elasticsearch/elasticsearch.log
and in journalctl -xe but I can't find anything that tells me the error.

Something else is taking memory I suspect... Perhaps Kibana

If you use the default Elasticsearch will claim 50% of the available RAM ... If it is not available it will crash.

To configure the heap size, add the Xms and Xmx JVM arguments to a custom JVM options file with the extension .options and store it in the jvm.options.d/ directory. For example, to set the maximum heap size to 2GB, set both Xms and Xmx to 2g:

In your case I would try 1GB

-Xms1g
-Xmx1g

Or get a bit larger machine, elasticsearch is greedy for RAM and does not really like other processes running unless you set the heap size

Thanks Stephen

I edited jvm.options and and modify the file with -Xms1g -Xmx1g
Now my free memory is: (include Kibana Up)

root@XXX:/etc/elasticsearch# free -h
total used free shared buff/cache available
Mem: 3,8Gi 2,9Gi 557Mi 0,0Ki 413Mi 760Mi
Swap: 0B 0B 0B

Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.