How to define multiline.pattern for complex pattern

Elastic Stack Beats
1

I need to define the multiline.pattern field in my prospectors, instead of doing it in the logstash filter.
The first pattern looks like that:
<Feb 24, 2016 4:30:07 PM IST>
The second pattern looks like that:
<<ERROR>> [Mar 01 10:05:16]
Before, In Logstash, I was using customized patterns.

For the first one:
pattern => "^\<%{WEBLOGICTIMESTAMP} " where in my patter file I define : WEBLOGICTIMESTAMP %{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} %{DL}

For the second one:pattern => "^\<\<%{LOGLEVEL}\>\> "

Thanks
Sharon.

2

What do the intermediate lines that you want to merge look like? Can you show a sample section of the file?

3

This is line from one log:

    <Feb 28, 2016 9:41:57 AM IST> <Error> <HTTP> <BEA-101020> <[ServletContext@836526773[app:ABPServer_abp.ear module:c3att path:null spec-version:3.0]] Servlet failed with an Exception
    java.lang.NullPointerException
            at jsp_servlet._rpl.__x1434944766_0_0.printInfoParam(__x1434944766_0_0.java:305)
            at jsp_servlet._rpl.__x1434944766_0_0._jspService(__x1434944766_0_0.java:610)
            at weblogic.servlet.jsp.JspBase.service(JspBase.java:35)
            at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
            at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
            Truncated. see log file for complete stacktrace
    >

This is line from second log:

<<ERROR>> [Feb 29 12:51:36] [[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] [CM] <PayChannelServicesBean.l3UpdatePcnBalanceExpirationIndicator> encountered an exception. This bean uses Container-Managed transactions. Hence, this flow will NOT be retried. Last retry count = <0 out of 0>

Thanks
Sharon

4

Just on a side note, if your looking to test your multi-line patterns, you can use the filebeat-multiline-tester tool. I haven't had much feedback on it yet, although it's been useful for me a bunch of times to test various patterns for filebeat configs.

5

If the patterns are always in separate files, can you not just simplify it by defining multiple prospectors?

6

yes, sure, I define two different prospector and I define multiline.pattern per each prospector.

The issue is to define the right multiline.pattern.

I was thinking about:

multiline.pattern: '^[[:graph:]][[:alpha:]][[:space:]][[:digit:]]{2}[[:graph:]][[:space:]][[:digit:]]{4}[[:space:]][[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}'

and

multiline.pattern: '^[[:graph:]][[:alpha:]][[:graph:]]'
       multiline.negate: true

accordingly

7

It is not clear to me how to install and run the tester.

I download and put the directory on my unix.

What now?

Thanks
Sharon.

8

still waiting for help here, how to define the pattern. Is what I did, is on the right direction? Should it be simpler?

9

Do the files only contain events with these patterns or are there also events that are not multiline? It would help if you could provide a sample from each file type, e.g in a gist.

10

Wow. Great. I will load some sample to the my github and send you the link.

Thanks,
Sharon.

11

Here are the log files examples:

https://gist.github.com/ssasporta/8776da18c72fbd23f907154e3bb83b54.js

12

Something looks wrong with this Gist. Can you try recreating it with just text files?

13
14

For the CMServer.log file it looks to me like every new event begins with a line starting with <<, so you should be able to build a pattern around this if that is the case.

For the ABPServer.log file it looks like every new event begins with a line starting with ####, so you should be able to build a pattern around this if that is the case.

The weblogic.log file seems more complicated and I am not sure how you would like the multiline events to be assembled.

I do unfortunately not have time to write and test this myself at the moment, but hopefully these pointers will get you started.

15

Still having problems opening the gist files.

Being lazy I just copied all content shown in this discussion and used this regex ^[#\<]

The trick about multiline is not looking at the content, but looking at the structure and re-occuring patterns at beginning or end of lines.

We do have a playground script for users to test multiline patterns. I adapted the script to include all your logs: https://play.golang.org/p/3Eneqg-oN5

Also check out the multiline tester: https://github.com/hartfordfive/filebeat-multiline-tester .
Find executables here: Releases

1 Like
16

great, as I have more logs and I will have to use this playground/tester.

Anyway, are you saying that the regex ^[#\<] fits all my various format? Should I put it in all the prospectors?

Currently the filebeat is failing to start. A log isn't even created in /var/log/filebeat. I am trying to understand what is wrong there.

If you have any direction for me, it will be perfect.

Thanks
Sharon.

17

For the starting: Can you try to run it with -e -d "*" and see if you get some output to the console?

18

I do not think it will work for your weblogic.log file, as you in that file seem to have lines that I suspect should be part of a multiline entry that begin with <. Maybe something like ^[\<][A-Z][a-z]{2}[[:space:]], which matches the first part of the timestamp, could work for this file, although I have not tested it.

19

Thanks. I will test it and let you know.

20

You are right. The weblogic one looks quite crazy. I'd definitely put weblogic.log into a separate prospector. weblogic.log always starts with <, plus date. You can try to be a little more strict by doing ^\<[JFMASOND][a-z]{2} \d{2}, or try something like ^\<[^\<]. The latter pattern matches a string starting with < not followed by another <. The pattern [^...] negates the characters listed.