Filebeat multiline is ignoring my settings

[eran.yo (Eran)]

Hi All,

I'm trying to use the filebeat multiline option to marge logs data that have header and footer.

My system logs have fixed header (#SQ#)and footer (#EOM#).
for example :

#SQ#|1551902277335100043|2019-03-06 21:57:57.335|INFO|0x00007f4570d1f700:c0s3|127.0.0.1|5000|0|master|qwerty11|3|qwerty11|32|"1"|#EOM#

While the log line fit to one line everything is OK and I gets one entry in Kibana.
But sometime when the log is long the system divide it to few lines, but the fixed header and footer are starting and ending the log, for example :

#SQ#|1551903090635451463|2019-03-06 22:11:30.634|INFO|0x00007f4570d1f700:c0s3|127.0.0.1|5000|0|master|qwerty11|4|qwerty11|1|"SELECT l_returnflag,
l_linestatus,
l_linestatus
ORDER BY l_returnflag,
l_linestatus
;"|#EOM#

But in this case in Kibaba ill see for each log line different entry .

I tried to set the multiline option in filebeat.yml as follow :

multiline.pattern: '#SQ#'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: '#EOM#'

But it doesn't meagre the all the log value between my header and footer values.

My ELK stack flow is : log file -->Filebeat-->Elasticelasticsearch-->Kibana

Can someone please help me and advice what i'm doing wrong?

Thanks,
Eran.

[pierhugues (Pier-Hugues Pellerin)]

Hello @eran.yo,

Can you try with the following options?

multiline.pattern: '^#SQ#'
multiline.negate: true

I don't think you need flush_pattern since you have a distinct beginning of event and the last event will be flush by the timeout.

I've used the following playground to test your options.

[eran.yo (Eran)]

Hi @pierhugues,

Thanks a lot for your help here.
This option solved my issue

Thanks,
Eran.