Logstash crashes when receiving a bad message

Elastic Stack Logstash
1

Hello Fellow Logstashians,

I am working on exporting Dynatrace data to SPLUNK via logstash. I am using the Dynatrace Real Time Business Transaction Feed and 2 instances of logstash with a redis queue in between. Every night, between 1am and 3am, the Logstash instance that receives the initial message from Dynatrace is being sent a message that causes it to crash with the following error.

2016-12-04 01:51:05 -0500: HTTP parse error, malformed request (): #<Puma::HttpParserError: Invalid HTTP format, parsing fails.>
2016-12-04 01:51:05 -0500: ENV: {"rack.version"=>[1, 3], "rack.errors"=>#<IO:fd 2>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "SCRIPT_NAME"=>"", "QUERY_STRING"=>"", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"2.16.0", "GATEWAY_INTERFACE"=>"CGI/1.2", "REQUEST_METHOD"=>"GET"}

LogStash::ConfigurationError: Only String and Array types are splittable. field:businessTransactions is of type = NilClass
filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-split-2.0.5/lib/logstash/filters/split.rb:49
multi_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/filters/base.rb:151
each at org/jruby/RubyArray.java:1613
multi_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/filters/base.rb:148
initialize at (eval):902
each at org/jruby/RubyArray.java:1613
initialize at (eval):898
call at org/jruby/RubyProc.java:281
filter_func at (eval):401
filter_batch at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:267
each at org/jruby/RubyArray.java:1613
inject at org/jruby/RubyEnumerable.java:852
filter_batch at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:265
worker_loop at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:223
start_workers at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:201

I have seen many different posts about similar issue and most of them end with a fix being applied to an earlier version of Logstash. I am running 2.4.

2

Also, this same setup runs for many many hours before receiving this death message.

3

You haven't showed us your configuration but it seems you should wrap your split filter in a

if [businessTransactions] { ... }

conditional.

4

Sorry, here is the piece of my configuration that is throwing the error. I believe it already has what you are suggesting:

if !("splitted_bts" in [tags] and [businessTransactions]) {
json {
source => "message"
}
split {
field => "businessTransactions"
add_tag => ["splitted_bts"]
}
}

5

No, you have a negation in your conditional. The json and split filters will process the event if there's no splitted_bts tag or businessTransactions is not set.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.