Logstash Not Connecting to Elasticsearch

Elastic Stack Logstash
1

Hello everyone,

I am running into a problem when I try to output Logstash to Elasticsearch. Everything works well when I'm merely outputting to stdout, but once I try to output to Elasticsearch, I am hit with this warning:

[`main]-pipeline-manager] WARN logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x54b107ff URL:http://localhost:9200/>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

Followed by this error:

ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::Elasticsearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}

Logstash then tries to connect to Elasticsearch indefinitely, but never gets a connection. I am able to connect to Elasticsearch independently of Logstash by accessing localhost:9200 on my machine. My output is included below.

output {
  elasticsearch
  {
    hosts => ["localhost:9200"]
  }
  stdout
  {
    codec => rubydebug
  }
}

Any advice would be appreciated. Thanks!

2

Hello there, this output config should just work. Is this a single node Elasticsearch setup you have? In what ways are you determining a successful ES connection independently?

3

Can you check whether Elasticsearch is currently accessible at "localhost:9200" ?

you should get something like this as shown below.

{
name: "ApsHWKB",
cluster_name: "elasticsearch",
cluster_uuid: "WdN9zzwSSzSq1n5PfepHPA",
version: {
number: "5.4.1",
build_hash: "2dfe0df",
build_date: "2017-05-29T16:05:51.443Z",
build_snapshot: false,
lucene_version: "6.5.1"
},
tagline: "You Know, for Search"
}

4

Yes, this is a single node setup that I am running with Docker. And I am verifying it by accessing "localhost:9200" and getting the standard output as shown below. In addition, I have a second index that I am working with on the same node that I built with elasticsearch.js, and I can interact with that index without any issues. However, for this index, I need the tools that Logstash provides.

{
"name" : "3PkNrHT",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "coHnZ2F5QtW2oal84BDAtQ",
"version" : {
"number" : "5.4.0",
"build_hash" : "780f8c4",
"build_date" : "2017-04-28T17:43:27.229Z",
"build_snapshot" : false,
"lucene_version" : "6.5.0"
},
"tagline" : "You Know, for Search"
}

5

@relder how are you testing that its working? Are you using curl http://localhost:9200? Logstash currently does not think it can access that URL.

6

I am using Postman to test, but essentially, yes. When I type in http://localhost:9200, I get this response:

{
"name": "TcGHLSD",
"cluster_name": "elasticsearch",
"cluster_uuid": "pZz4iIQ2Tc6r_oa5n7ubuA",
"version": {
"number": "5.4.0",
"build_hash": "780f8c4",
"build_date": "2017-04-28T17:43:27.229Z",
"build_snapshot": false,
"lucene_version": "6.5.0"
},
"tagline": "You Know, for Search"
}

7

This is bizarre. This is really one of the most common codepaths in Logstash. I'm thinking something is somehow different with networking from the command line?

I can't repro this myself.

8

It turns out the issue was with the way I had Docker configured rather than any issue with Logstash. It was connecting to the localhost of the Docker virtual machine rather than the localhost of my main machine thus it wasn't able to find my Elasticsearch container. For anyone who thinks they might be running into a similar problem, add the command --net=host to the line that launches your container. Thanks for the help everyone.

1 Like
9

I am also stuck with same error .. In my case I have a cluster of 3 nodes .. At first logstash creates connection with all nodes of elasticsearch but then it shows connection refused error .
Here is the trash .

[2017-06-16T11:16:52,689][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@master:9200/, :path=>"/"}
[2017-06-16T11:16:52,704][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x566c5ec1 URL:http://elastic:xxxxxx@master:9200/>}
[2017-06-16T11:16:52,705][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@slave-1:9200/, :path=>"/"}
[2017-06-16T11:16:52,714][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x275b256b URL:http://elastic:xxxxxx@slave-1:9200/>}
[2017-06-16T11:16:52,714][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@slave-2:9200/, :path=>"/"}
[2017-06-16T11:16:52,722][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x465e163b URL:http://elastic:xxxxxx@slave-2:9200/>}
[2017-06-16T11:16:52,723][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-06-16T11:16:52,786][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-06-16T11:16:52,792][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>[#<URI::Generic:0x2240e641 URL://master>, #<URI::Generic:0x7787fa5a URL://slave-1>, #<URI::Generic:0x72773861 URL://slave-2>]}
[2017-06-16T11:16:52,837][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/home/mywavia/Downloads/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.0.4-java/vendor/GeoLite2-City.mmdb"}
[2017-06-16T11:16:52,848][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-06-16T11:16:52,861][INFO ][logstash.pipeline ] Pipeline main started
The stdin plugin is now waiting for input:
[2017-06-16T11:16:52,897][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-06-16T11:16:57,634][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-16T11:16:57,641][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x21878895 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-16T11:16:57,656][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-16T11:16:57,660][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x35b76f7 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-16T11:17:02,644][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}

Can anyone help with this ?

10

What are you using to run your instances of Logstash and Elasticsearch?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.