Multiline codec

Elastic Stack Logstash
1

Hi,

I'm using multiline codec in my input configuration in order to merge lines in one "message".
This lines are from a tomcat log. But as most of times it works, others doesn't. It mix logs from other "non-tomcat" logs from other servers (i guess).
Why is mixing that? Is there a way to avoid this behaviour? I don't know if it's a best practise to use filter instead of input plugin.

Thanks

2

Please show your input configuration.

3

Hi Magnus,

input {
lumberjack {
port => 5043
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"

    codec => multiline {
            pattern => "(^%{TIMESTAMP_ISO8601} ) | (^%{IPV4} )"
            negate => true
            what => "previous"
            }
    }

    udp {
    port => 25826
    codec => collectd {}
    type => collectd
    }

}

4

Yeah, you can't really use the multiline codec for inputs that read events from multiple sources like lumberjack does. Switch to Filebeat and use it's recently introduced multiline feature.

5

Ahhh, ok Magnus.

What about using multiline filter? is a good option?

Thanks

6

The multiline filter is deprecated. That said, with its stream_identity option you should be able to use it.

7

Thanks Magnus,

However, i think that my two patterns are ot working.
I mean: if i receive a IP address it stills adding the line to the previous message, which doesn't happen if i receive a timestamp.
What's the way to user several patterns (A or B or C...)

Thank you!

8

Start by changing

pattern => "(^%{TIMESTAMP_ISO8601} ) | (^%{IPV4} )"

to

pattern => "(^%{TIMESTAMP_ISO8601})|(^%{IPV4})"
9

Hi Magnus,

It works.
But finally i moved to beats (as you recomended).
What happens now is that this patterns (that i used in logstash multifilter) don't work for filebeat...
Doesn't the patterns be the same?

Thanks again

10

You're not showing your configuration so I'll have to guess: Filebeat's multiline support doesn't support the grok patterns that Logstash's multiline codec supports. You'll have to use standard regular expressions.

11

Hi,

the patter i writed above.

multiline:

    pattern: "(^%{TIMESTAMP_ISO8601})|(^%{IPV4})"
    negate: true
    match: after
    max_lines: 200
    timeout: 4s
  tail_files: true
12

As I suspected you're trying to use grok patterns. Use a standard regexp instead.

13

Hello,

I'm trying with regexp but with no luck.
Filebeats is not sending any data dute to multiline.
I tried with this pattern
^([0-9]{4})(-?)(1[0-2]|0[1-9])(-?)(3[01]|0[1-9]|[12][0-9])\s(2[0-3]|[01][0-9]):?([0-5][0-9]):?([0-5][0-9])$
Just for the timestamp

14

I reply myself,

The problem was de \s which i changed for [[:space:]] however it multilines all mesages, even if the pattern is within the message...
so i guess my pattern is not correct... no idea why.

The data i receiv is YYYY-MM-dd HH:mm:ss

Any idea?

15

Please show your complete multiline configuration and an example message. Format everything as code.