Hi i have elk 8.5.1 installed on a machine with 8 gb ram and 50 gb hardisk. i've set jvm options to 2g for elasticsearch and 2g for logstash. I've elk stack +filebeat installed here. and i am getting logs from jenkins, nexus, apache, elk, maven ,zabbix using filebeat on each of these servers .
Everything works fine. but elasticsearch service fails again and again. When i restart it it works fine for a while and fails again. Here is the log
Nov 23 10:52:39 ip-10-0-9-223 logstash: [2022-11-23T10:52:39,494][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused"}
Nov 23 10:52:44 ip-10-0-9-223 kibana: [2022-11-23T10:52:44.494+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 10.0.9.223:9200 error
Nov 23 10:52:44 ip-10-0-9-223 logstash: [2022-11-23T10:52:44,497][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused>}
Nov 23 10:52:44 ip-10-0-9-223 logstash: [2022-11-23T10:52:44,497][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused"}
Nov 23 10:52:49 ip-10-0-9-223 logstash: [2022-11-23T10:52:49,501][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused>}
Nov 23 10:52:49 ip-10-0-9-223 logstash: [2022-11-23T10:52:49,501][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused"}
Nov 23 10:52:49 ip-10-0-9-223 filebeat: {"log.level":"info","@timestamp":"2022-11-23T10:52:49.935Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":38500},"total":{"ticks":939280,"time":{"ms":10},"value":939280},"user":{"ticks":900780,"time":{"ms":10}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":17},"info":{"ephemeral_id":"75357dfd-dce0-4b57-853c-7a7c5458e0c1","uptime":{"ms":20130639},"version":"8.5.1"},"memstats":{"gc_next":1867247752,"memory_alloc":934228032,"memory_total":121491323904,"rss":1154785280},"runtime":{"goroutines":92}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":2}},"output":{"events":{"active":600},"read":{"bytes":36}},"pipeline":{"clients":7,"events":{"active":4101}}},"registrar":{"states":{"current":17}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
Nov 23 10:52:50 ip-10-0-9-223 kibana: [2022-11-23T10:52:50.603+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 10.0.9.223:9200 error
Nov 23 10:52:54 ip-10-0-9-223 logstash: [2022-11-23T10:52:54,504][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused>}
Nov 23 10:52:54 ip-10-0-9-223 logstash: [2022-11-23T10:52:54,505][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused"}
Nov 23 10:52:58 ip-10-0-9-223 logstash: [2022-11-23T10:52:58,931][ERROR][logstash.outputs.elasticsearch][main][7f4ba291cce5f998e906f3936f25e9ca4c329e0dbc0b7c39be6038e0bcae90d8] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>64}
Nov 23 10:52:59 ip-10-0-9-223 logstash: [2022-11-23T10:52:59,508][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused>}
Nov 23 10:52:59 ip-10-0-9-223 logstash: [2022-11-23T10:52:59,508][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused"}