Elasticsearch Unreachable error in Logstash

Elastic Stack Logstash
1

Hi,

I have just upgraded from ELK 5.0.0 to 5.2.0. I have installed and set X-Pack 5.2.0 for Elasticsearch, Logstash and Kibana, and enabled monitoring alone in Kibana and Elasticsearch. Kibana and Elasticsearch are working fine. I have enabled a sample Logstash file to read from Kafka, but getting the below trace;

[2017-02-15T13:04:09,029][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200, :path=>"/"}
[2017-02-15T13:04:09,035][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x2186e33b URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200][Manticore::SocketException] Connection refused"}

Somebody please kind enough to tell me why is this happening and How can I fix this. Also, I want to use an IP 192.168.1.100 instead of localhost here. How can I do this?

Thanks in advance.

2 Likes
2

here is the entire trace;

Sending Logstash's logs to data/GA5.2/logstash-5.2.0/logs which is now configured via log4j2.properties
[2017-02-15T13:44:06,720][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s]}}
[2017-02-15T13:44:06,725][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200, :path=>"/"}
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2017-02-15T13:44:06,863][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x16168399 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200][Manticore::SocketException] Connection refused"}
[2017-02-15T13:44:06,864][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::HTTP:0x4e703f0d URL:http://localhost:9200>]}
[2017-02-15T13:44:06,865][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2}
[2017-02-15T13:44:06,875][INFO ][logstash.pipeline        ] Pipeline .monitoring-logstash started
[2017-02-15T13:44:07,008][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-02-15T13:44:07,036][INFO ][logstash.pipeline        ] Pipeline main started
[2017-02-15T13:44:07,124][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-02-15T13:44:11,878][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200, :path=>"/"}
[2017-02-15T13:44:11,889][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x721fec92 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200][Manticore::SocketException] Connection refused"}
[2017-02-15T13:44:16,545][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s][Manticore::SocketException] Connection refused {:url=>http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s, :error_message=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s][Manticore::SocketException] Connection refused", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2017-02-15T13:44:16,546][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s][Manticore::SocketException] Connection refused", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2017-02-15T13:44:16,891][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200, :path=>"/"}
[2017-02-15T13:44:16,896][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x604dcabe URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200][Manticore::SocketException] Connection refused"}
[2017-02-15T13:44:18,550][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
3

Somebody please kind enough to tell me why is this happening and How can I fix this.

Can you connect to localhost:9200 yourself? Is that where Elasticsearch is supposed to be running?

Also, I want to use an IP 192.168.1.100 instead of localhost here. How can I do this?

Change the URL in your Logstash configuration to say 192.168.1.100 instead of localhost?

1 Like
4

Hi @magnusbaeck

Can you connect to localhost:9200 yourself? Is that where Elasticsearch is supposed to be running?

No, I cant. I use a remote machine with just console access and i always have to use 192.168.1.100 instead of localhost, and it works fine

Change the URL in your Logstash configuration to say 192.168.1.100 instead of localhost?

Where is that setting? I use logstash to output just to console alone, using the setting stdout{ codec => json }

5

Seeing this as well, I would love a minor update that doesn't break things...
Another good idea is to have a single place in which to set parameters, regardless of whether init or systemd is in use, to be honest it's getting a bit pathetic, just trying to launch logstash as a different user requires changing a systemd file.

For reference, nothing at all referring to localhost in the logstash conf dir:

[root@log01 /etc/logstash] $ grep -ir localhost /etc/logstash/ .

Seeing this perpetually repeating in the logs:

[2017-02-16T14:51:18,055][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-02-16T14:51:18,059][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x172c387f URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-02-16T14:51:23,066][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-02-16T14:51:23,071][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xfdd5d4c URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-02-16T14:51:28,073][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-02-16T14:51:28,077][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x3a615a3e URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-02-16T14:51:33,078][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-02-16T14:51:33,083][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x56490e06 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-02-16T14:51:38,084][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-02-16T14:51:38,088][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x4d8ab7be URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

Version installed:
[root@log01 /etc/logstash] $ yum info logstash
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile

2 Likes
7

OK reading through the docs a bit more as I'm also hit with this issue, it seems that the initial setup NEGLECTS to mention that there are xpack settings for logstash.yml that includes the elasticsearch url for monitoring!!
https://www.elastic.co/guide/en/x-pack/current/monitoring-settings.html#monitoring-logstash-settings

I'm just testing this now.

4 Likes
8

BINGO!!

[2017-02-28T10:19:24,022][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.1.1.89:9200/, :path=>"/"}
[2017-02-28T10:19:24,029][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x1a2e77f7 URL:http://10.1.1.89:9200/>}

You MUST add the following config lines to your /etc/logstash/logstash.yml

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://X.X.X.X:9200

Where X.X.X.X is the IP of your Elastic cluster.

EDIT: Clarity

23 Likes
9

Thanks! Now it's working fine. Maybe i'm going to deactivate the monitoring later but for now (testing purposes) it is fine!

1 Like
10

Thank you!

I've been sweating this problem for a few hours - it seems to have fixed logstash w/ x-pack for me as well.

2 Likes
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.