I have two identical VM where ELK 6.6.2 is running, they are independent and not configured as a cluster.
On one Elasticsearch is running fine, on the other Elasticsearch crashes a few minutes after starting.
Here is the OS version.
uname -a
Linux XXXXXXXXXX 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Here is the ELK version.
/usr/share/elasticsearch/bin/elasticsearch --version
Version: 6.6.2, Build: default/rpm/3bd3e59/2019-03-06T15:16:26.864148Z, JVM: 1.8.0_201
/usr/share/kibana/bin/kibana --version
6.6.2
/usr/share/logstash/bin/logstash --version
logstash 6.6.2
In messages I can see
Mar 22 18:48:05 XXXXXXXXXX systemd: Started Elasticsearch.
Mar 22 18:59:28 XXXXXXXXXX logstash: [2022-03-22T18:59:28,054][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://X.X.X.X:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://X.X.X.X:9200/, :error_message=>"Elasticsearch Unreachable: [http://X.X.X.X:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
Mar 22 18:59:28 XXXXXXXXXX logstash: [2022-03-22T18:59:28,053][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://X.X.X.X:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://X.X.X.X:9200/, :error_message=>"Elasticsearch Unreachable: [http://X.X.X.X:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
Mar 22 18:59:28 XXXXXXXXXX logstash: [2022-03-22T18:59:28,054][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://X.X.X.X:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
Mar 22 18:59:28 XXXXXXXXXX logstash: [2022-03-22T18:59:28,055][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://X.X.X.X:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
Mar 22 18:59:30 XXXXXXXXXX logstash: [2022-03-22T18:59:30,060][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
Mar 22 18:59:30 XXXXXXXXXX logstash: [2022-03-22T18:59:30,063][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
Mar 22 18:59:31 XXXXXXXXXX systemd: elasticsearch.service: main process exited, code=killed, status=6/ABRT
Mar 22 18:59:31 XXXXXXXXXX systemd: Unit elasticsearch.service entered failed state.
Mar 22 18:59:31 XXXXXXXXXX kibana: {"type":"log","@timestamp":"2022-03-22T17:59:31Z","tags":["error","elasticsearch","admin"],"pid":7688,"message":"Request error, retrying\nGET http://localhost:9200/_nodes/_local?filter_path=nodes.*.settings.tribe => read ECONNRESET"}
Mar 22 18:59:31 XXXXXXXXXX systemd: elasticsearch.service failed.
A file has been generated hs_err_pid6489.log.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f0510c3bb52, pid=6489, tid=0x00007f050c89a700
#
# JRE version: OpenJDK Runtime Environment (8.0_201-b09) (build 1.8.0_201-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.201-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x692b52]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00007f0508116800): VMThread [stack: 0x00007f050c79b000,0x00007f050c89b000] [id=6913]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000040
The full file : 1 https://pastebin.com/F2vrpJFg, 2 https://pastebin.com/DFPfUVyU, 3 https://pastebin.com/9sJ38cS0
I have compared the Elasticsearch config files between both VM they are the same.
Can anyone help find a solution?