Dear Team,
On our kibana server elasticsearch service stopping automatically and when we are trying to start it ,its showing error as below :
[root@ELKMONP3 ~]# systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
------>> Attaching the journalctl -xe logs below :
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/sbin/OSWatcher: fork: retry: Resource temporarily unavailable
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/ifconfigsub: fork: retry: No child processes
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/xtop: fork: retry: No child processes
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:40 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:41 ELKMONP3 logstash[13636]: [2024-03-03T08:58:41,512][ERROR][logstash.outputs.elasticsearch][statuslog][1c073bf4627230e59a0ebcdbbaac4fca94fbd5ccd54659cd98
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/sbin/OSWatcher: fork: retry: Resource temporarily unavailable
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/ifconfigsub: fork: retry: No child processes
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/xtop: fork: retry: No child processes
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:41 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,065][WARN ][logstash.outputs.elasticsearch][statuslog] Attempted to resurrect connection to dead ES insta
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,128][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,128][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,128][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,131][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,132][ERROR][logstash.outputs.elasticsearch][maindms][ac92c407e067a12caa8de7477485909ddb1f25726d9f180dc933
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,483][ERROR][logstash.outputs.elasticsearch][certiconsoletrans][256ad69b2f9ed146e00e773424cf9f941debae3431
Mar 03 08:58:42 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/iosub: fork: retry: Resource temporarily unavailable
Mar 03 08:58:42 ELKMONP3 logstash[13636]: [2024-03-03T08:58:42,629][WARN ][logstash.outputs.elasticsearch][certiconsoletrans] Attempted to resurrect connection to dead
Mar 03 08:58:43 ELKMONP3 logstash[13636]: [2024-03-03T08:58:43,078][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:43 ELKMONP3 logstash[13636]: [2024-03-03T08:58:43,079][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:43 ELKMONP3 logstash[13636]: [2024-03-03T08:58:43,080][ERROR][logstash.outputs.elasticsearch][dmserrorcs][a1aed40f84d1046d1fcb860e763712ed6d70ff8e8e9f3f745
Mar 03 08:58:43 ELKMONP3 logstash[13636]: [2024-03-03T08:58:43,082][ERROR][logstash.outputs.elasticsearch][dmsconsolecs][8c360b543e1c6cdf5f8413b53a7551412fa27e81d9d96e9
Mar 03 08:58:43 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/iosub: fork: retry: Resource temporarily unavailable
Mar 03 08:58:43 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:43 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/oswsub: fork: retry: No child processes
Mar 03 08:58:43 ELKMONP3 OSWatcher[1323]: /usr/libexec/oswatcher/xtop: fork: retry: No child processes
------------------------------------------------------
When we checked Elasticsearch logs below attached error observed on server :