Multiline codec

Hi,

I'm using multiline codec in my input configuration in order to merge lines in one "message".
This lines are from a tomcat log. But as most of times it works, others doesn't. It mix logs from other "non-tomcat" logs from other servers (i guess).
Why is mixing that? Is there a way to avoid this behaviour? I don't know if it's a best practise to use filter instead of input plugin.

Thanks

Please show your input configuration.

Hi Magnus,

input {
lumberjack {
port => 5043
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"

    codec => multiline {
            pattern => "(^%{TIMESTAMP_ISO8601} ) | (^%{IPV4} )"
            negate => true
            what => "previous"
            }
    }
    udp {
    port => 25826
    codec => collectd {}
    type => collectd
    }

}

Yeah, you can't really use the multiline codec for inputs that read events from multiple sources like lumberjack does. Switch to Filebeat and use it's recently introduced multiline feature.

Ahhh, ok Magnus.

What about using multiline filter? is a good option?

Thanks

The multiline filter is deprecated. That said, with its stream_identity option you should be able to use it.

Thanks Magnus,

However, i think that my two patterns are ot working.
I mean: if i receive a IP address it stills adding the line to the previous message, which doesn't happen if i receive a timestamp.
What's the way to user several patterns (A or B or C...)

Thank you!

Start by changing

pattern => "(^%{TIMESTAMP_ISO8601} ) | (^%{IPV4} )"

to

pattern => "(^%{TIMESTAMP_ISO8601})|(^%{IPV4})"

Hi Magnus,

It works.
But finally i moved to beats (as you recomended).
What happens now is that this patterns (that i used in logstash multifilter) don't work for filebeat...
Doesn't the patterns be the same?

Thanks again

You're not showing your configuration so I'll have to guess: Filebeat's multiline support doesn't support the grok patterns that Logstash's multiline codec supports. You'll have to use standard regular expressions.

Hi,

the patter i writed above.

multiline:

    pattern: "(^%{TIMESTAMP_ISO8601})|(^%{IPV4})"
    negate: true
    match: after
    max_lines: 200
    timeout: 4s
  tail_files: true

As I suspected you're trying to use grok patterns. Use a standard regexp instead.

Hello,

I'm trying with regexp but with no luck.
Filebeats is not sending any data dute to multiline.
I tried with this pattern
^([0-9]{4})(-?)(1[0-2]|0[1-9])(-?)(3[01]|0[1-9]|[12][0-9])\s(2[0-3]|[01][0-9]):?([0-5][0-9]):?([0-5][0-9])$
Just for the timestamp

I reply myself,

The problem was de \s which i changed for [[:space:]] however it multilines all mesages, even if the pattern is within the message...
so i guess my pattern is not correct... no idea why.

The data i receiv is YYYY-MM-dd HH:mm:ss

Any idea?

Please show your complete multiline configuration and an example message. Format everything as code.