# Elasticsearch Down

**URL:** <https://discuss.elastic.co/t/elasticsearch-down/330175>\
**Category:** Elasticsearch\
**Created:** [April 18, 2023, 1:43am UTC](https://discuss.elastic.co/t/elasticsearch-down/330175 "2023-04-18T01:43:25Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lelc79](https://avatars.discourse-cdn.com/v4/letter/l/43a26b/32.png) [@Lelc79](https://discuss.elastic.co/u/Lelc79)\
**Post date:** [April 18, 2023, 1:43am UTC](https://discuss.elastic.co/t/elasticsearch-down/330175/1 "2023-04-18T01:43:25Z")

</div>

hi community  
My elasticsearch Server is Down, ¿what could be happening?

Elasticsearch 8.7

**1) systemctl status elasticsearch**

elasticsearch.service - Elasticsearch  
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)  
Active: failed (Result: oom-kill) since Mon 2023-04-17 20:14:58 -05; 13min ago  
Docs: [https://www.elastic.co](https://www.elastic.co)  
Process: 72295 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID\_DIR}/elasticsearch.pid --quiet (code=exited, status=143)  
Main PID: 72295 (code=exited, status=143)  
CPU: 19min 45.197s

Starting Elasticsearch...  
Started Elasticsearch.  
elasticsearch.service: A process of this unit has been killed by the OOM killer.  
ERROR: Elasticsearch exited unexpectedly  
elasticsearch.service: Failed with result 'oom-kill'.  
elasticsearch.service: Consumed 19min 45.197s CPU time.

**2) Logs /var/log/elasticsearch elasticsearch.log**

root@XXX:/var/log/elasticsearch# tail -f elasticsearch.log  
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[?:?]  
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[?:?]  
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[?:?]  
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]  
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]  
... 16 more  
[2023-04-17T20:14:45,952][WARN][o.e.h.n.Netty4HttpServerTransport] [XXX] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.0.1.20:9200, remoteAddress=/192.0.1.20:59818}  
[2023-04-17T20:14:47,674][WARN][o.e.h.n.Netty4HttpServerTransport] [XXX] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.0.1.20:9200, remoteAddress=/192.0.1.20:59822}  
[2023-04-17T20:14:48,398][WARN][o.e.h.n.Netty4HttpServerTransport] [XXX] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.0.1.20:9200, remoteAddress=/192.0.1.20:59832}  
[2023-04-17T20:14:50,570][WARN][o.e.h.n.Netty4HttpServerTransport] [XXX] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.0.1.20:9200, remoteAddress=/192.0.1.20:59840}

1. **journalctl -xe**

abr 17 20:39:28 XXX kernel: [73840] 1000 73840 3682 308 69632 1 0 sshd  
abr 17 20:39:28 XXX kernel: [73841] 1000 73841 2042 359 49152 28 0 bash  
abr 17 20:39:28 XXX kernel: [73847] 1000 73847 2502 146 65536 4 0 su  
abr 17 20:39:28 XXX kernel: [73848] 0 73848 1831 150 49152 3 0 bash  
abr 17 20:39:28 XXX kernel: [73849] 0 73849 32706 164 184320 6 0 journalctl  
abr 17 20:39:28 XXX kernel: [73850] 0 73850 1461 56 57344 4 0 less  
abr 17 20:39:28 XXX kernel: [73991] 0 73991 3630 307 73728 0 0 sshd  
abr 17 20:39:28 XXX kernel: [73997] 1000 73997 3682 309 73728 0 0 sshd  
abr 17 20:39:28 XXX kernel: [73998] 1000 73998 2042 386 57344 0 0 bash  
abr 17 20:39:28 XXX kernel: [74001] 1000 74001 2502 149 57344 0 0 su  
abr 17 20:39:28 XXX kernel: [74002] 0 74002 1819 149 53248 0 0 bash  
abr 17 20:39:28 XXX kernel: [74025] 0 74025 2535 68 57344 0 0 systemctl  
abr 17 20:39:28 XXX kernel: [74026] 0 74026 3294 149 65536 0 0 systemd-tty-ask  
abr 17 20:39:28 XXX kernel: [74027] 107 74027 650623 591 430080 17051 0 java  
abr 17 20:39:28 XXX kernel: [74086] 107 74086 1170357 470781 4648960 72371 0 java  
abr 17 20:39:28 XXX kernel: [74112] 107 74112 22876 130 77824 0 0 controller  
abr 17 20:39:28 XXX kernel: oom-kill:constraint=CONSTRAINT\_NONE,nodemask=(null),cpuset=/,mems\_allowed=0,global\_oom,task\_memcg=/system.slice/elasticsearch.service,task=java,pid=7408\>  
abr 17 20:39:28 XXX kernel: Out of memory: Killed process 74086 (java) total-vm:4681428kB, anon-rss:1883124kB, file-rss:0kB, shmem-rss:0kB, UID:107 pgtables:4540kB oom\_score\_adj:0  
abr 17 20:39:28 XXX kibana[72442]: [2023-04-17T20:39:26.012-05:00][WARN][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: co\>  
abr 17 20:39:28 XXX kibana[72442]: [2023-04-17T20:39:26.642-05:00][WARN][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: co\>  
abr 17 20:39:30 XXX systemd-entrypoint[74027]: ERROR: Elasticsearch exited unexpectedly  
abr 17 20:39:30 XXX systemd[1]: elasticsearch.service: Main process exited, code=exited, status=137/n/a  
░░ Subject: Unit process exited  
░░ Defined-By: systemd  
░░ Support: [Debian -- User Support](https://www.debian.org/support)  
░░  
░░ An ExecStart= process belonging to unit elasticsearch.service has exited.  
░░  
░░ The process' exit code is 'exited' and its exit status is 137.  
abr 17 20:39:30 XXX systemd[1]: elasticsearch.service: Failed with result 'exit-code'.  
░░ Subject: Unit failed  
░░ Defined-By: systemd  
░░ Support: [Debian -- User Support](https://www.debian.org/support)  
░░  
░░ The unit elasticsearch.service has entered the 'failed' state with result 'exit-code'.  
abr 17 20:39:30 XXX systemd[1]: Failed to start Elasticsearch.  
░░ Subject: A start job for unit elasticsearch.service has failed  
░░ Defined-By: systemd  
░░ Support: [Debian -- User Support](https://www.debian.org/support)  
░░  
░░ A start job for unit elasticsearch.service has finished with a failure.  
░░  
░░ The job identifier is 11055 and the job result is failed.  
abr 17 20:39:30 XXX systemd[1]: elasticsearch.service: Consumed 55.067s CPU time.  
░░ Subject: Resources consumed by unit runtime  
░░ Defined-By: systemd  
░░ Support: [Debian -- User Support](https://www.debian.org/support)  
░░  
░░ The unit elasticsearch.service completed and consumed the indicated resources.

**4) Please help me ...**

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 18, 2023, 1:49am UTC](https://discuss.elastic.co/t/elasticsearch-down/330175/2 "2023-04-18T01:49:16Z")

</div>

> [@Lelc79](#):
>
> Active: failed (Result: oom-kill)

Looks like It ran out of memory / RAM and died.

What size server RAM is it on and what else is running on the server?

Did you manually set the JVM heap size or just used the defaults.

---

<div class="post-metadata">

**Author:** ![Lelc79](https://avatars.discourse-cdn.com/v4/letter/l/43a26b/32.png) [@Lelc79](https://discuss.elastic.co/u/Lelc79)\
**Post date:** [April 18, 2023, 2:05am UTC](https://discuss.elastic.co/t/elasticsearch-down/330175/3 "2023-04-18T02:05:41Z")

</div>

My test computer has 2CPU and 4GB of RAM, It only runs ELasticSearch and Kibana. it has the default JVM heap size setting. It is a new configuration.

**Free -h**  
total used free shared buff/cache available  
Mem: 3,8Gi 1,9Gi 1,8Gi 0,0Ki 120Mi 1,8Gi  
Swap: 0B 0B 0B

**systemctl status elasticsearch**

● elasticsearch.service - Elasticsearch  
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)  
Active: failed (Result: exit-code) since Mon 2023-04-17 22:18:54 -05; 1min 37s ago  
Docs: [https://www.elastic.co](https://www.elastic.co)  
Process: 75680 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID\_DIR}/elasticsearch.pid --quiet (code=exited, status=137)  
Main PID: 75680 (code=exited, status=137)  
CPU: 7.997s

abr 17 22:18:45 XXX systemd[1]: Starting Elasticsearch...  
abr 17 22:18:53 XXX systemd-entrypoint[75680]: ERROR: Elasticsearch exited unexpectedly  
abr 17 22:18:54 XXX systemd[1]: elasticsearch.service: Main process exited, code=exited, status=137/n/a  
abr 17 22:18:54 XXX systemd[1]: elasticsearch.service: Failed with result 'exit-code'.  
abr 17 22:18:54 XXX systemd[1]: Failed to start Elasticsearch.  
abr 17 22:18:54 XXX systemd[1]: elasticsearch.service: Consumed 7.997s CPU time.

Is there an easy way to view the logs?

I am checking in **/var/elasticsearch/elasticsearch.log**  
and in **journalctl -xe** but I can't find anything that tells me the error.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 18, 2023, 3:27am UTC](https://discuss.elastic.co/t/elasticsearch-down/330175/4 "2023-04-18T03:27:38Z")

</div>

Something else is taking memory I suspect... Perhaps Kibana

If you use the default Elasticsearch will claim 50% of the available RAM ... If it is not available it will crash.

> **[Important Elasticsearch configuration | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#heap-size-settings)**

> To configure the heap size, add the Xms and Xmx JVM arguments to a custom JVM options file with the extension .options and store it in the jvm.options.d/ directory. For example, to set the maximum heap size to 2GB, set both Xms and Xmx to 2g:

In your case I would try 1GB

```auto
-Xms1g
-Xmx1g

```

Or get a bit larger machine, elasticsearch is greedy for RAM and does not really like other processes running unless you set the heap size

---

<div class="post-metadata">

**Author:** ![Lelc79](https://avatars.discourse-cdn.com/v4/letter/l/43a26b/32.png) [@Lelc79](https://discuss.elastic.co/u/Lelc79)\
**Post date:** [April 18, 2023, 12:48pm UTC](https://discuss.elastic.co/t/elasticsearch-down/330175/5 "2023-04-18T12:48:19Z")

</div>

Thanks Stephen

I edited jvm.options and and modify the file with -Xms1g -Xmx1g  
Now my free memory is: (include Kibana Up)

root@XXX:/etc/elasticsearch# free -h  
total used free shared buff/cache available  
Mem: 3,8Gi 2,9Gi 557Mi 0,0Ki 413Mi 760Mi  
Swap: 0B 0B 0B

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 16, 2023, 12:48pm UTC](https://discuss.elastic.co/t/elasticsearch-down/330175/6 "2023-05-16T12:48:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
