Multiline codec is not working with filebeats plugin in logstash 2.0.0

Elastic Stack Beats
1

Multiline codec is not working with filebeats plugin (version 0.9.3) in logstash 2.0.0, were as logstash-forwarder plugin works perfectly fine. Here is the code that I am having problem with:
input { beats { port => 5043 type => "someinput" codec => multiline { max_lines => 2000 pattern => "\d{10}\.\d{6}" negate => true what => "previous" } }

Anyone having similar issues with multiline codec in beats plugin?

Thanks.

2

I would expect this to work, since this part of the code is similar to the lumberjack input.
If you use the stdout output with the rubydebug codec what do you see?

What is the incorrect behavior? The events aren't merged or you don't get any output?

3

Yes, I do get stdout output, but without any stdout codec. BTW, rubydebug codec crashes when used with filebeats input.

But it is not merging the events correctly based on the provided pattern. It is behaving in an unpredictable fashion. Events are merged at odd locations. Oddly enough, it actually eats up the matched pattern itself from the input, but merges multiple chunks of the input between matched patterns in one long event.

Here is the error from stdout ruby codec:

  NoMethodError: undefined method `to_hash' for ["  119"]:Array
  encode_default at /home/fdps/ELK/logstash-2.0.0.latest/vendor/bundle/jruby/1.9/gems/logstash-codec-rubydebug-2.0.3/lib/logstash/codecs/rubydebug.rb:34
            call at org/jruby/RubyMethod.java:120
          encode at /home/fdps/ELK/logstash-2.0.0.latest/vendor/bundle/jruby/1.9/gems/logstash-codec-rubydebug-2.0.3/lib/logstash/codecs/rubydebug.rb:30
         receive at /home/fdps/ELK/logstash-2.0.0.latest/vendor/bundle/jruby/1.9/gems/logstash-output-stdout-2.0.2/lib/logstash/outputs/stdout.rb:54
          handle at /home/fdps/ELK/logstash-2.0.0.latest/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/outputs/base.rb:80
     output_func at (eval):22
    outputworker at /home/fdps/ELK/logstash-2.0.0.latest/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:252
   start_outputs at /home/fdps/ELK/logstash-2.0.0.latest/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:169
4

Is is possible to see your filebeat configuration?

5

I am able to reproduce your issue! I am checking where its located

6

Just released logstash-input-beats 0.9.6 that fix this issue

You can update the plugin with this command bin/plugin update logstash-input-beats

7

I have the same issue. logstash 2.0, logstash-input-beats 0.9.6

input {
beats {
port => 5044
codec => multiline {
pattern => "(^.*\sPHP\s((Stack trace)|(\s\s\d)|(\s\d))|^\s|^'|^Stack trace|^#)"
what => previous
negate => false
}
}
}

" it actually eats up the matched pattern itself from the input, but merges multiple chunks of the input between matched patterns in one long event."