Elasticsearch service stopping automatically after each 2-3 days

[2328943_dc (Vandana)]

Dear Team,

On our kibana server elasticsearch service stopping automatically and when we are trying to start it ,its showing error as below :

[root@ELKMONP3 ~]# systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.


------>> Attaching the journalctl -xe logs below  :  
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/sbin/OSWatcher: fork: retry: Resource temporarily unavailable
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/ifconfigsub: fork: retry: No child processes
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/xtop: fork: retry: No child processes
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:41 ELKMONP3 logstash[13636]: [2024-03-03T08:58:41,512][ERROR][logstash.outputs.elasticsearch][statuslog][1c073bf4627230e59a0ebcdbbaac4fca94fbd5ccd54659cd98
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/sbin/OSWatcher: fork: retry: Resource temporarily unavailable
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/ifconfigsub: fork: retry: No child processes
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/xtop: fork: retry: No child processes
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,065][WARN ][logstash.outputs.elasticsearch][statuslog] Attempted to resurrect connection to dead ES insta
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,128][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,128][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,128][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,131][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,132][ERROR][logstash.outputs.elasticsearch][maindms][ac92c407e067a12caa8de7477485909ddb1f25726d9f180dc933
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,483][ERROR][logstash.outputs.elasticsearch][certiconsoletrans][256ad69b2f9ed146e00e773424cf9f941debae3431
Mar 03 08:58:42 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/iosub: fork: retry: Resource temporarily unavailable
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,629][WARN ][logstash.outputs.elasticsearch][certiconsoletrans] Attempted to resurrect connection to dead
Mar 03 08:58:43 ELKMONP3 logstash[13636]: [2024-03-03T08:58:43,078][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:43 ELKMONP3 logstash[13636]: [2024-03-03T08:58:43,079][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:43 ELKMONP3 logstash[13636]: [2024-03-03T08:58:43,080][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:43 ELKMONP3 logstash[13636]: [2024-03-03T08:58:43,082][ERROR][logstash.outputs.elasticsearch][dmsconsolecs][8c360b543e1c6cdf5f8413b53a7551412fa27e81d9d96e9
Mar 03 08:58:43 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/iosub: fork: retry: Resource temporarily unavailable
Mar 03 08:58:43 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:43 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:43 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/xtop: fork: retry: No child processes
------------------------------------------------------

When we checked Elasticsearch logs below attached error observed on server :

[stephenb (Stephen Brown)]

Hi @2328943_dc

Please don't paste images of text on the future.

To get the journalctl for elasticsearch please run

journalctl --unit elasticsearch -f

Also the imaged
shows elasticsearch is OOM i.e.is ran Out Of Memory.

What version are you running?

Plus it looks like you are running logstash on the same sever which can cause resource contention and us not considered best practice.

Elasticsearch will try to claim 50% of the RAM as JVM heap.. if it can not it will crash with OOM. This is not uncommon with resource contention

You can set the JVM max heap if needed.

Important Elasticsearch configuration | Elasticsearch Guide [8.12] | Elastic

Advanced configuration | Elasticsearch Guide [8.12] | Elastic

What are the server specs and what all else are you running? Kibana , logstash and elasticsearch?

[rtwolfe94022 (Tim Wolfe)]

@stephenb is correct on resource issues

The errors indicate your system might be running out of resources, causing Elasticsearch to stop. Check for high resource usage, ensure system limits are sufficient, and review Elasticsearch's configuration and logs for more specific errors. Adjusting system limits and verifying cluster health can also help resolve the issue.