Filebeat - multiline pattern does not work as expected

[mateusz-lubanski-omn (Mateusz Lubanski)]

Hi,

I set up multiline.* properties for /etc/filebeat/filebeat.yml as bellow:
filebeat.inputs:
- type: log
paths:
- /home/eip/logs/*.log
tail_files: true
multiline.pattern: '^%{TIMESTAMP_ISO8601}'
multiline.negate: true
multiline.match: after

Here sample how above configuration group logs

Unfortunatelly filebeat does not group lines as expected and is groupping much more than is should.
Is there maybe another configuration which cause the issue or is value of multiline.pattern property defined wrongly?

Thanks Mateusz

[shaunak (Shaunak Kashyap)]

Could you post a sample log file and an example of your desired grouping here? For posting the sample log file you might need to use something like https://pastebin.com/ if it's too large. Once I have these I can do some experimentation to try and come up with the right multiline settings.

[mateusz-lubanski-omn (Mateusz Lubanski)]

Hi @shaunak

Sure, let me please provide more details.
For logging we are using log4j with ConversattionPattern: %d [%t] %-5p - %m%n
So sample loog look like: https://pastebin.com/V1NnxMQc

whenever line in log file starts with pattern "%d [%t] %-5p" defined by log4j filebeat should group next lines till pattern occur again. Expected multiline logs will look like:

• https://pastebin.com/5qwB2pYY
• https://pastebin.com/Jr19cJjP
• https://pastebin.com/ez7ZNUJD

Thanks, Mateusz

[mateusz-lubanski-omn (Mateusz Lubanski)]

Below pattern solved problem:
multiline.pattern: '^\d{4}-\d{2}-\d{2} '