Beats Multiline Pattern Error

Elastic Stack Beats
1

I am trying to ingest multiline log event with filebeat and using Multiline Pattern but it is not working out as expected. Here is one of the multiline event I have:

03 Aug 2020 02:39:53,456 DEBUG [sometext] [sometext] ResourceStatusReader:220 - Persisted Resource Event:

EventKOK:
   id                 = ResourceStatus-kk0000_opiii-2020-08-03T02-39-52Z
   description        = Resource Update Event
   type               = Comment
   start              = Aug 3, 2020 2:39:52 AM [965270392 0 [2020/08/03 02:39:52.000]]
   stop               = Aug 3, 2020 2:39:52 AM [965270392 0 [2020/08/03 02:39:52.000]]
   resource names     = [kk0000_opiii]
   params         = 
Name: Status	Value: OK

and the regex I have for the pattern:

^(?:\d{2}\s[\w]{3}\s\d{4}\s(\d{2}:?){3},\d{3})\s+[A-Z]{4,7}\s+\[.+\]\s+\[.+\]\s+.+\n+^EventKOK:

This does not give me one event. But makes it into two separate events. The first one before the return line:

03 Aug 2020 02:39:53,456 DEBUG [sometext] [sometext] ResourceStatusReader:220 - Persisted Resource Event:

and the other one:

EventKOK:
   id                 = ResourceStatus-kk0000_opiii-2020-08-03T02-39-52Z
   description        = Resource Update Event
   type               = Comment
   start              = Aug 3, 2020 2:39:52 AM [965270392 0 [2020/08/03 02:39:52.000]]
   stop               = Aug 3, 2020 2:39:52 AM [965270392 0 [2020/08/03 02:39:52.000]]
   resource names     = [kk0000_opiii]
   params         = 
Name: Status	Value: OK

Thank you for your help.

2

It seems you would like to partition your events based on the timestamp at the start of the line, right? If so, how about removing this part from the end of your multiline regex: \n+^EventKOK:?

Shaunak

3

Thank you Shaunak. I tried excluding and including\n+^EventKOK:On both cases, filebeat consider it as two different events and raised parsing error for the part starting EventKOK: id = ResourceStatus-kk0000_opiii-2020-08-03T02-39-52Z description = Resource Update Event type = Comment start = Aug 3, 2020 2:39:52 AM [965270392 0 [2020/08/03 02:39:52.000]] stop = Aug 3, 2020 2:39:52 AM [965270392 0 [2020/08/03 02:39:52.000]] resource names = [kk0000_opiii] params = Name: Status Value: OK The whole event, starting with timestamp 03 Aug 2020 02:39:53,456 to the last line, .Name: Status Value: OK should be in one event.

4

I just tested your sample log input with the following Filebeat multiline configuration and it seemed to work as expected:

  multiline:
    pattern: '^\d{2}\s\w{3}\s\d{4}'
    negate: true
    match: after

Could you use the above regex?

Shaunak

5

I see the problem. I have many other multiline patterns which work fine with,

    negate:  false
    match:  before

When I changed the configuration, it worked now.
Thank you.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.