LogStash runner error: An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>

bizmate · October 5, 2018, 12:45am

I am trying to run LogStash on my machine through a docker container.
I am setting up the env variables to

HTTP_HOST=https://myawses.us-east-1.es.amazonaws.com:443/
HTTP_PORT=443

Also all the outputs i have in my configurations are going to this url on AWS ie
output {
elasticsearch {
hosts => ["https://myawses.us-east-1.es.amazonaws.com:443"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

or

output {
elasticsearch {
hosts => ["https://myawses.us-east-1.es.amazonaws.com:443"]
index => "myindex"
}
stdout { codec => rubydebug }
}

But not only i still see the default elasticsearch:9200 host in the logs

logstash_1 | [2018-10-05T00:24:59,049][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}

but also it fails with message

logstash_1          | [2018-10-05T00:25:08,134][INFO ][logstash.inputs.metrics  ] Monitoring License OK
logstash_1          | [2018-10-05T00:25:08,185][INFO ][org.logstash.beats.Server] Starting server on port: 5044
logstash_1          | [2018-10-05T00:25:09,074][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add_tcp_listener'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:88:in `start_webserver'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:44:in `block in run'", "org/jruby/RubyRange.java:485:in `each'", "org/jruby/RubyEnumerable.java:1067:in `each_with_index'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:39:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:353:in `block in start_webserver'"]}
logstash_1          | [2018-10-05T00:25:09,164][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

So i verified the configuration and disabled XPACK with the following

bash-4.2$ cat /usr/share/logstash/config/logstash.yml
http.host: https://search-bizmate....k4sm.us-east-1.es.amazonaws.com:443/
http.port: 443
xpack.monitoring.elasticsearch.url: https://search-bizmate...k4sm.us-east-1.es.amazonaws.com:443/
xpack.monitoring.enabled: false

But i am still getting error

logstash_1          | [2018-10-06T22:23:29,755][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add_tcp_listener'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:88:in `start_webserver'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:44:in `block in run'", "org/jruby/RubyRange.java:485:in `each'", "org/jruby/RubyEnumerable.java:1067:in `each_with_index'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:39:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:353:in `block in start_webserver'"]}
logstash_1          | [2018-10-06T22:23:29,837][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

And this error is strange, my service is active, running and accessible from my host and it is on AWS. I can also curl it from the container and get a good response.

{
name: "4O...n",
cluster_name: "50994...7:biz....es",
cluster_uuid: "6oT.........qZQ",
version: {
number: "6.3.1",
build_flavor: "oss",
build_type: "zip",
build_hash: "e6df130",
build_date: "2018-08-01T07:23:36.254471Z",
build_snapshot: false,
lucene_version: "7.3.1",
minimum_wire_compatibility_version: "5.6.0",
minimum_index_compatibility_version: "5.0.0"
},
tagline: "You Know, for Search"
}

So I am really not sure what is happening

axrayn · October 8, 2018, 12:58am

Hi Diego,

The error is coming up when Logstash is trying to start the internal API server and it fails because it can't resolve "https://myawses.us-east-1.es.amazonaws.com" to an IP address (because of the 'HTTPS://' part).

Change your env variable to:

HTTP_HOST=myawses.us-east-1.es.amazonaws.com

If you already have a service running on port 443 you won't be able to specify the same port for HTTP_PORT but as this is just for the metrics api, I'd leave it as the default port and not specify the HTTP_PORT variable.

Hope this helps.

bizmate · October 8, 2018, 2:00am

Hi Axrayn

thank you for your suggestion. I had already tried. Indeed I think I have tried almost anything. Based on your suggestion I removed the https:// prefix and I removed the HTTP_POST env variable (I am using docker containers) and this in turn would remove it from the generic configuration. Also XPACK_MONITORING_ENABLED=false

In the conf.d configuration I also change the output sections to

output {
  elasticsearch {
    hosts => ["myawses.us-east-1.es.amazonaws.com:443"]

And now i get

logstash_1          | 2018/10/08 01:49:53 Setting 'http.host' from environment.
logstash_1          | 2018/10/08 01:49:53 Setting 'xpack.monitoring.enabled' from environment.
logstash_1          | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1          | [2018-10-08T01:50:59,238][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash_1          | [2018-10-08T01:50:59,266][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash_1          | [2018-10-08T01:51:00,353][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"69eba438-f345-4bb8-8a84-afcfbe4c96ef", :path=>"/usr/share/logstash/data/uuid"}
logstash_1          | [2018-10-08T01:51:01,215][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
logstash_1          | [2018-10-08T01:51:09,442][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
logstash_1          | [2018-10-08T01:51:11,683][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://myawses.us-east-1.es.amazonaws.com:443/]}}
logstash_1          | [2018-10-08T01:51:11,695][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://myawses.us-east-1.es.amazonaws.com:443/, :path=>"/"}
logstash_1          | [2018-10-08T01:51:12,232][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://myawses.us-east-1.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '400' contacting Elasticsearch at URL 'http://myawses.us-east-1.es.amazonaws.com:443/'"}
logstash_1          | [2018-10-08T01:51:12,258][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//myawses.us-east-1.es.amazonaws.com:443"]}
logstash_1          | [2018-10-08T01:51:12,283][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://myawses.us-east-1.es.amazonaws.com:443/]}}
logstash_1          | [2018-10-08T01:51:12,284][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://myawses.us-east-1.es.amazonaws.com:443/, :path=>"/"}
logstash_1          | [2018-10-08T01:51:12,512][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://myawses.us-east-1.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '400' contacting Elasticsearch at URL 'http://myawses.us-east-1.es.amazonaws.com:443/'"}
logstash_1          | [2018-10-08T01:51:12,514][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//myawses.us-east-1.es.amazonaws.com:443"]}
logstash_1          | [2018-10-08T01:51:13,852][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
logstash_1          | [2018-10-08T01:51:13,913][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x40e1709b run>"}
logstash_1          | [2018-10-08T01:51:14,232][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1          | [2018-10-08T01:51:14,425][INFO ][org.logstash.beats.Server] Starting server on port: 5044
logstash_1          | [2018-10-08T01:51:14,849][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add_tcp_listener'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:88:in `start_webserver'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:44:in `block in run'", "org/jruby/RubyRange.java:485:in `each'", "org/jruby/RubyEnumerable.java:1067:in `each_with_index'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:39:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:353:in `block in start_webserver'"]}
logstash_1          | [2018-10-08T01:51:14,908][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

As you can see logstash doesnt use https unless explicitly set. Is the above change in configuration what your suggestion was about?

bizmate · October 8, 2018, 5:27pm

Just a quick update as this is now solved. I had to remove all the configuration from the default logstash configuration, ie this file /usr/share/logstash/config/logstash.yml has now been left with default values.
Also, in the outputs of the configurations, I had to use https:// for the protocol and :433 at the end to ensure it was not defaulting to http and 9200 defaults

system · November 5, 2018, 5:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
An unexpected error occurred! SocketError: initialize: name or service not known Logstash	3	1607	December 27, 2020
Error: initialize: name or service not known Logstash	5	5382	August 27, 2018
LogStash-Outputs -ElasticSearch - URI is not valid - host is not specified Logstash docker	2	4111	April 6, 2020
Incorrect ELASTIC_HOSTS variable in Logstash container Logstash docker	2	122	April 15, 2024
Error [Manticore::ResolutionFailure] elasticsearch: Name or service not known"} at docker-compose Elasticsearch	3	6480	July 3, 2019

LogStash runner error: An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>

But i am still getting error

Related topics