LogStash runner error: An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>


(Diego Gullo) #1

I am trying to run LogStash on my machine through a docker container.
I am setting up the env variables to

  • HTTP_HOST=https://myawses.us-east-1.es.amazonaws.com:443/
  • HTTP_PORT=443

Also all the outputs i have in my configurations are going to this url on AWS ie
output {
elasticsearch {
hosts => ["https://myawses.us-east-1.es.amazonaws.com:443"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

or

output {
elasticsearch {
hosts => ["https://myawses.us-east-1.es.amazonaws.com:443"]
index => "myindex"
}
stdout { codec => rubydebug }
}

But not only i still see the default elasticsearch:9200 host in the logs

logstash_1 | [2018-10-05T00:24:59,049][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}

but also it fails with message

logstash_1          | [2018-10-05T00:25:08,134][INFO ][logstash.inputs.metrics  ] Monitoring License OK
logstash_1          | [2018-10-05T00:25:08,185][INFO ][org.logstash.beats.Server] Starting server on port: 5044
logstash_1          | [2018-10-05T00:25:09,074][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add_tcp_listener'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:88:in `start_webserver'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:44:in `block in run'", "org/jruby/RubyRange.java:485:in `each'", "org/jruby/RubyEnumerable.java:1067:in `each_with_index'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:39:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:353:in `block in start_webserver'"]}
logstash_1          | [2018-10-05T00:25:09,164][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

So i verified the configuration and disabled XPACK with the following

bash-4.2$ cat /usr/share/logstash/config/logstash.yml
http.host: https://search-bizmate....k4sm.us-east-1.es.amazonaws.com:443/
http.port: 443
xpack.monitoring.elasticsearch.url: https://search-bizmate...k4sm.us-east-1.es.amazonaws.com:443/
xpack.monitoring.enabled: false

But i am still getting error

logstash_1          | [2018-10-06T22:23:29,755][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add_tcp_listener'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:88:in `start_webserver'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:44:in `block in run'", "org/jruby/RubyRange.java:485:in `each'", "org/jruby/RubyEnumerable.java:1067:in `each_with_index'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:39:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:353:in `block in start_webserver'"]}
logstash_1          | [2018-10-06T22:23:29,837][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

And this error is strange, my service is active, running and accessible from my host and it is on AWS. I can also curl it from the container and get a good response.

{
name: "4O...n",
cluster_name: "50994...7:biz....es",
cluster_uuid: "6oT.........qZQ",
version: {
number: "6.3.1",
build_flavor: "oss",
build_type: "zip",
build_hash: "e6df130",
build_date: "2018-08-01T07:23:36.254471Z",
build_snapshot: false,
lucene_version: "7.3.1",
minimum_wire_compatibility_version: "5.6.0",
minimum_index_compatibility_version: "5.0.0"
},
tagline: "You Know, for Search"
}

So I am really not sure what is happening


(Axrayn) #2

Hi Diego,

The error is coming up when Logstash is trying to start the internal API server and it fails because it can't resolve "https://myawses.us-east-1.es.amazonaws.com" to an IP address (because of the 'HTTPS://' part).

Change your env variable to:

If you already have a service running on port 443 you won't be able to specify the same port for HTTP_PORT but as this is just for the metrics api, I'd leave it as the default port and not specify the HTTP_PORT variable.

Hope this helps.


(Diego Gullo) #3

Hi Axrayn

thank you for your suggestion. I had already tried. Indeed I think I have tried almost anything. Based on your suggestion I removed the https:// prefix and I removed the HTTP_POST env variable (I am using docker containers) and this in turn would remove it from the generic configuration. Also XPACK_MONITORING_ENABLED=false

In the conf.d configuration I also change the output sections to

output {
  elasticsearch {
    hosts => ["myawses.us-east-1.es.amazonaws.com:443"]

And now i get

logstash_1          | 2018/10/08 01:49:53 Setting 'http.host' from environment.
logstash_1          | 2018/10/08 01:49:53 Setting 'xpack.monitoring.enabled' from environment.
logstash_1          | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1          | [2018-10-08T01:50:59,238][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash_1          | [2018-10-08T01:50:59,266][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash_1          | [2018-10-08T01:51:00,353][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"69eba438-f345-4bb8-8a84-afcfbe4c96ef", :path=>"/usr/share/logstash/data/uuid"}
logstash_1          | [2018-10-08T01:51:01,215][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
logstash_1          | [2018-10-08T01:51:09,442][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
logstash_1          | [2018-10-08T01:51:11,683][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://myawses.us-east-1.es.amazonaws.com:443/]}}
logstash_1          | [2018-10-08T01:51:11,695][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://myawses.us-east-1.es.amazonaws.com:443/, :path=>"/"}
logstash_1          | [2018-10-08T01:51:12,232][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://myawses.us-east-1.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '400' contacting Elasticsearch at URL 'http://myawses.us-east-1.es.amazonaws.com:443/'"}
logstash_1          | [2018-10-08T01:51:12,258][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//myawses.us-east-1.es.amazonaws.com:443"]}
logstash_1          | [2018-10-08T01:51:12,283][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://myawses.us-east-1.es.amazonaws.com:443/]}}
logstash_1          | [2018-10-08T01:51:12,284][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://myawses.us-east-1.es.amazonaws.com:443/, :path=>"/"}
logstash_1          | [2018-10-08T01:51:12,512][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://myawses.us-east-1.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '400' contacting Elasticsearch at URL 'http://myawses.us-east-1.es.amazonaws.com:443/'"}
logstash_1          | [2018-10-08T01:51:12,514][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//myawses.us-east-1.es.amazonaws.com:443"]}
logstash_1          | [2018-10-08T01:51:13,852][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
logstash_1          | [2018-10-08T01:51:13,913][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x40e1709b run>"}
logstash_1          | [2018-10-08T01:51:14,232][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1          | [2018-10-08T01:51:14,425][INFO ][org.logstash.beats.Server] Starting server on port: 5044
logstash_1          | [2018-10-08T01:51:14,849][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add_tcp_listener'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:88:in `start_webserver'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:44:in `block in run'", "org/jruby/RubyRange.java:485:in `each'", "org/jruby/RubyEnumerable.java:1067:in `each_with_index'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:39:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:353:in `block in start_webserver'"]}
logstash_1          | [2018-10-08T01:51:14,908][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

As you can see logstash doesnt use https unless explicitly set. Is the above change in configuration what your suggestion was about?


Adding patterns in ES on AWS, pipeline definition through API
Adding patterns in ES on AWS, pipeline definition through API
(Diego Gullo) #4

Just a quick update as this is now solved. I had to remove all the configuration from the default logstash configuration, ie this file /usr/share/logstash/config/logstash.yml has now been left with default values.
Also, in the outputs of the configurations, I had to use https:// for the protocol and :433 at the end to ensure it was not defaulting to http and 9200 defaults


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.