Elasticsearch Unreachable or Down!

cjs6891 · August 2, 2017, 5:15pm

Hi,

My ELK stack has been running great for months, until recently. I've been receiving the following error:

[2017-08-01T13:02:10,092][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-01T13:02:10,092][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-01T13:05:06,207][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message
=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::ClientProtocolException] localhost:9200 failed to respond", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool
::HostUnreachableError", :will_retry_in_seconds=>2}
[2017-08-01T13:05:06,207][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message
=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection reset", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError",
:will_retry_in_seconds=>2}
[2017-08-01T13:05:07,171][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perha
ps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_re
try_in_seconds=>2}

My configuration hasn't change since November, any advice or direction would be appreciated

Julien · August 2, 2017, 5:55pm

the error mentions elasticsearch on localhost is not listening or responding and you have not specified what appear in ES logs or used cluster health url to verify if ES cluster is up
Can you check in the logs of elasticsearch if and why elasticsearch is stopped?
Also a few unrelated general recommendations that would probably have prevented the issue
1)you should run logstash separate from ES cluster as both can use a lot of cpu resources.
2)You should also have more than one node for ES cluster in which case logstash can use the other ES nodes when one node is not accessible

cjs6891 · August 2, 2017, 7:02pm

I am, and always have been running my ELK Stack on a single node...

all I'm catching in my stdout & stderr is what I've posted

I'll get the stack up & running, even digesting new logs and it randomly has been crashing for the last few days

load average is only 5.12 on a 24 core system, memory consumption is only 4GB of 64GB

is there a limit to the number of indexes, I have about 305

cjs6891 · August 2, 2017, 7:29pm

log [19:10:35.005] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow
log [19:10:40.453] [error][status][plugin:elasticsearch@5.0.0] Status changed from yellow to red - Elasticsearch is still initializing the kibana index.
log [19:10:40.455] [error][status][ui settings] Status changed from yellow to red - Elasticsearch plugin is red

[2017-08-02T15:14:58,566][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$6@6f1b56b3 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52a2ea7f[Running, pool size = 24, active threads = 24, queued tasks = 50, completed tasks = 45381]]"})
[2017-08-02T15:14:58,566][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$6@6f1b56b3 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52a2ea7f[Running, pool size = 24, active threads = 24, queued tasks = 50, completed tasks = 45381]]"})
[2017-08-02T15:14:58,566][ERROR][logstash.outputs.elasticsearch] Retrying individual actions

[2017-08-02T15:15:15,675][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$6@23827dd0 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52a2ea7f[Running, pool size = 24, active threads = 20, queued tasks = 50, completed tasks = 47986]]"})
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Retrying individual actions
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,675][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,676][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,676][ERROR][logstash.outputs.elasticsearch] Action
[2017-08-02T15:15:15,676][ERROR][logstash.outputs.elasticsearch] Action
log [19:24:54.688] [info][status][plugin:elasticsearch@5.0.0] Status changed from red to green - Kibana index ready
log [19:24:54.691] [info][status][ui settings] Status changed from red to green - Ready
log [19:24:57.379] [error][elasticsearch] Request error, retrying
HEAD http://localhost:9200/ => read ECONNRESET
[2017-08-02T15:24:57,381][WARN ][logstash.outputs.elasticsearch] Marking url as dead. {:reason=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection reset", :url=>#<URI::HTTP:0x245abf1e URL:http://localhost:9200>, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection reset", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2017-08-02T15:24:57,391][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection reset", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
log [19:24:57.406] [warning][elasticsearch] Unable to revive connection: http://localhost:9200/
log [19:24:57.406] [warning][elasticsearch] No living connections
log [19:24:57.410] [error][status][plugin:elasticsearch@5.0.0] Status changed from green to red - Unable to connect to Elasticsearch at http://localhost:9200.
log [19:24:57.411] [error][status][ui settings] Status changed from green to red - Elasticsearch plugin is red

Julien · August 3, 2017, 9:55am

These seem to be logstash logs which just specify ES is not reachable on localhost:9200... Which you can check via curl -XGET http://localhost:9200

elasticsearch*.log files are located in the "logs" folder of the elasticsearch installation folder. If ES is down, restarting it might be the easy solution though you will need to check why it is down from the logs

cjs6891 · August 3, 2017, 12:36pm

Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/ELK_Storage/Elasticsearch_Index_Storage/nodes/0/indices/lLv1lYnqQsmuL8DkQxhclQ/2/index/_7i.fnm") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 8270 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]

I start elasticsearch as follows:

**# ulimit -l unlimited**
**# /sbin/runuser elk -c 'ES_JAVA_OPTS="-Xms12g -Xmx12g" /opt/ELK_5.0/elasticsearch-5.0.0/bin/elasticsearch -p /opt/ELK_5.0/elasticsearch-5.0.0/tmp/elasticsearch-pid -d' 1> /tmp/elasticsearch_stdout 2> /tmp/elasticsearch_stderr &**

cjs6891 · August 3, 2017, 1:13pm

ulimit -v & ulimit -m both returned ‘unlimited’. I doubled vm.max_map_count, so far so good...

Julien · August 3, 2017, 2:47pm

Please use the recommended value as a minimum :

cjs6891 · August 4, 2017, 12:43pm

@Julien thank you, I appreciate it

system · September 1, 2017, 12:43pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.