Logstash FATAL error when cannot connect to elasticsearch - v6.2.3

[sunny12 (sunny)]

Hi there,
Thanking in advance for any replies!!

Symptom: If logstash cannot connect to elasticsearch, logstash has a fatal error and shutdown completely.

Expected behavior: If cannot connect to elasticsearch, or there a configuration error, logstash would log the error but not exit fatally

Logstash version: 6.2.3

To simulate.

1. Setup logstash. Run logstash as a service.
2. Configure elasticsearch in output plugin
3. Do not setup elasticsearch. Ensure that logstash has never been able to connect to elasticsearch.

[2018-04-24T17:31:21,819][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method <' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:213:inget_event_type'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:165:in event_action_params'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:39:inevent_action_tuple'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:34:in block in multi_receive'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:34:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:479:inblock in output_batch'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:478:inoutput_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:430:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385:inblock in start_workers'"]}

[2018-04-24T17:53:02,393][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit

Any help is greatly appreciated.

[JKhondhu (Jymit Singh Khondhu)]

What is in your logstash.yml and share the full logstash.conf used please.
Is Elasticsearch running in your scenario above?

With what you shared you should have:

[WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error.......:error=>"Elasticsearch Unreachable: [...........9200/][Manticore::SocketException] Connection refused (Connection refused)"}

and no fatal exit.

[sunny12 (sunny)]

Ok we were able to find a resolution.

The resolution is to set document_type => doc in the elasticsearch output configuration

Reason (hypothesis):
Logstash plugin connects to elasticsearch to determine the elasticsearch version and sets the event type accordingly. Link : https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/common.rb

If logstash was able to connect to elasticsearch at least once, logstash knows the elasticsearch version and adjusts accordingly.

However, logstash was never able to connect to elasticsearch ever, it exists with FATAL error. Even non-elasticsearch output pipelines are not processed.

Setting document_type => doc basically tells logstash the elasticsearch version is >= 6
so logstash does not need at least one successful connection to elasticsearch

logstash.yml

path.data: "/var/lib/logstash"
path.config: "/etc/logstash/conf.d"
path.logs: "/var/log/logstash"
config.reload.automatic: false

logstash.conf (without document_type)

input {
file {
path => ["/var/log/messages"]
type => "syslog"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
if [type] == "syslog" {
elasticsearch {
hosts => [ "http://xxxx-xxx:9200" ]
index => "xxx"
user => "xxx"
password => "xxxx"
}
}
}