I have a regular problem of logstash crashing with FATAL on Linux RHEL 6.9. The logstash version is 6.6.1.
The pipeline.yml contains following config :
- pipeline.id: sys_username_mapping
pipeline.workers: 1
path.config: "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/config/sys_username_mapping/logstash-sys_username_mapping.conf" - pipeline.id: sys_useremail_mapping
pipeline.workers: 1
path.config: "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/config/sys_useremail_mapping/logstash-sys_useremail_mapping.conf" - pipeline.id: ps_lic_server_mapping
pipeline.workers: 1
path.config: "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/config/ps_lic_server_mapping/logstash-servergoup.conf"
Here is the logstash crash log snippet. Please help to understand the root cause
[2019-08-28T08:35:13,185][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"", :_index=>"sys_useremail_mapping", :_type=>"doc", :routing=>nil}, #LogStash::Event:0x5015e843], :response=>{"index"=>{"_index"=>"sys_useremail_mapping", "_type"=>"doc", "_id"=>"", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"if _id is specified it must not be empty"}}}}
[2019-08-28T08:35:13,186][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"", :_index=>"sys_useremail_mapping", :_type=>"doc", :routing=>nil}, #LogStash::Event:0x8a006bf], :response=>{"index"=>{"_index"=>"sys_useremail_mapping", "_type"=>"doc", "_id"=>"", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"if _id is specified it must not be empty"}}}}
[2019-08-28T09:01:25,235][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<SystemCallError: Unknown error (SystemCallError) - >, :backtrace=>["org/jruby/RubyIO.java:1457:in write'", "org/jruby/RubyIO.java:1428:inwrite'", "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-stdout-3.1.4/lib/logstash/outputs/stdout.rb:43:in block in multi_receive_encoded'", "org/jruby/RubyArray.java:1734:ineach'", "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-stdout-3.1.4/lib/logstash/outputs/stdout.rb:42:in multi_receive_encoded'", "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/logstash-core/lib/logstash/outputs/base.rb:87:inmulti_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:inmulti_receive'", "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:390:in block in output_batch'", "org/jruby/RubyHash.java:1343:ineach'", "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:341:inworker_loop'", "/pa/vertica_backup_qa/otherscripts/elk/logstash/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:304:in `block in start_workers'"]}
[2019-08-28T09:01:25,370][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit