I am trying to run LogStash on my machine through a docker container.
I am setting up the env variables to
- HTTP_HOST=https://myawses.us-east-1.es.amazonaws.com:443/
- HTTP_PORT=443
Also all the outputs i have in my configurations are going to this url on AWS ie
output {
elasticsearch {
hosts => ["https://myawses.us-east-1.es.amazonaws.com:443"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
or
output {
elasticsearch {
hosts => ["https://myawses.us-east-1.es.amazonaws.com:443"]
index => "myindex"
}
stdout { codec => rubydebug }
}
But not only i still see the default elasticsearch:9200 host in the logs
logstash_1 | [2018-10-05T00:24:59,049][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
but also it fails with message
logstash_1 | [2018-10-05T00:25:08,134][INFO ][logstash.inputs.metrics ] Monitoring License OK
logstash_1 | [2018-10-05T00:25:08,185][INFO ][org.logstash.beats.Server] Starting server on port: 5044
logstash_1 | [2018-10-05T00:25:09,074][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add_tcp_listener'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:88:in `start_webserver'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:44:in `block in run'", "org/jruby/RubyRange.java:485:in `each'", "org/jruby/RubyEnumerable.java:1067:in `each_with_index'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:39:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:353:in `block in start_webserver'"]}
logstash_1 | [2018-10-05T00:25:09,164][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
So i verified the configuration and disabled XPACK with the following
bash-4.2$ cat /usr/share/logstash/config/logstash.yml
http.host: https://search-bizmate....k4sm.us-east-1.es.amazonaws.com:443/
http.port: 443
xpack.monitoring.elasticsearch.url: https://search-bizmate...k4sm.us-east-1.es.amazonaws.com:443/
xpack.monitoring.enabled: false
But i am still getting error
logstash_1 | [2018-10-06T22:23:29,755][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add_tcp_listener'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:88:in `start_webserver'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:44:in `block in run'", "org/jruby/RubyRange.java:485:in `each'", "org/jruby/RubyEnumerable.java:1067:in `each_with_index'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:39:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:353:in `block in start_webserver'"]}
logstash_1 | [2018-10-06T22:23:29,837][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
And this error is strange, my service is active, running and accessible from my host and it is on AWS. I can also curl it from the container and get a good response.
{
name: "4O...n",
cluster_name: "50994...7:biz....es",
cluster_uuid: "6oT.........qZQ",
version: {
number: "6.3.1",
build_flavor: "oss",
build_type: "zip",
build_hash: "e6df130",
build_date: "2018-08-01T07:23:36.254471Z",
build_snapshot: false,
lucene_version: "7.3.1",
minimum_wire_compatibility_version: "5.6.0",
minimum_index_compatibility_version: "5.0.0"
},
tagline: "You Know, for Search"
}
So I am really not sure what is happening