Logstash stopped processing because of an error: (SystemExit) exit

[alexus]

Hello World!

I'm Running Logstash on Docker | Logstash Reference [6.8] | Elastic and getting following error:

logstash    | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash    | [2019-08-15T20:50:25,185][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["gelf", "nre", "upsertDome", "upsertMaid", "upsertMinion"], :poll_interval=>"5000000000ns"}
logstash    | [2019-08-15T20:50:26,012][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@localhost:9200/]}}
logstash    | [2019-08-15T20:50:26,323][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash    | [2019-08-15T20:50:26,385][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://elastic:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused) {:url=>https://elastic:xxxxxx@localhost:9200/, :error_message=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash    | [2019-08-15T20:50:26,398][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash    | [2019-08-15T20:50:26,585][ERROR][logstash.configmanagement.elasticsearchsource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
logstash    | [2019-08-15T20:50:26,598][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.>, :backtrace=>["/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:67:in `with_license_check'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:46:in `initialize'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:41:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:71:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:304:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
logstash    | [2019-08-15T20:50:26,612][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

there is no error if I'm using logstash 6.8.0 (w/ same configuration), yet error seems to appear only if I use logstash 6.8.1 or 6.8.2.

Please advise.

[alexus]

ERROR: Unable to retrieve license information from license server

Hello World! I'm Running Logstash on Docker | Logstash Reference [6.8] | Elastic, yet running into following error: logstash | [2019-09-07T15:15:06,686][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["gelf", "nre", "upsertDome", "upsertMaid", "upsertMinion"], :poll_interval=>"5000000000ns"} logstash | [2019-09-07T15:15:07,936][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :adde…

[Tek_Chand]

@alexus

@alexus,

logstash | [2019-08-15T20:50:26,398][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

From the above error its seems that your logstash server is unable to communicate with elasticsearch server over port 9200. Please try to telnet elasticsearch_machine_ip 9200 from logstash machine. If unable to telnet then open the required firewall rule.

Thanks.

[alexus]

as soon as I roll back to 6.8.0 logstash is able to communicate with rest of cluster, it's only with 6.8.1+ it is unable to communicate, so the issue is not on my end, it is with logstash 6.8.1+, same thing I mentioned with my previous topic ...

[Tek_Chand]

@alexus, May be some issue with logstash 6.8.1+, Please check at your end again some things like are you using same version of logstash, Elasticsearch, firewall rule etc. I have tested logstash 6.4.0 and logstash 7.2.0.

Thanks.

[alexus]

there is definitely looks like an issue w/ logstash 6.8.1+

elasticsearch cluster is running latest 6.8.3 version, yet logstash exits w/ any version (from 6 branch) higher than 6.8.0, "firewall rules" are okay, as logstash works w/ 6.8.0.