How to define multiline.pattern for complex pattern


(Sharon Sasporta) #1

I need to define the multiline.pattern field in my prospectors, instead of doing it in the logstash filter.
The first pattern looks like that:
<Feb 24, 2016 4:30:07 PM IST>
The second pattern looks like that:
<<ERROR>> [Mar 01 10:05:16]
Before, In Logstash, I was using customized patterns.

For the first one:
pattern => "^\<%{WEBLOGICTIMESTAMP} " where in my patter file I define : WEBLOGICTIMESTAMP %{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} %{DL}

For the second one:pattern => "^\<\<%{LOGLEVEL}\>\> "

Thanks
Sharon.


(Christian Dahlqvist) #2

What do the intermediate lines that you want to merge look like? Can you show a sample section of the file?


(Sharon Sasporta) #3

This is line from one log:

    <Feb 28, 2016 9:41:57 AM IST> <Error> <HTTP> <BEA-101020> <[ServletContext@836526773[app:ABPServer_abp.ear module:c3att path:null spec-version:3.0]] Servlet failed with an Exception
    java.lang.NullPointerException
            at jsp_servlet._rpl.__x1434944766_0_0.printInfoParam(__x1434944766_0_0.java:305)
            at jsp_servlet._rpl.__x1434944766_0_0._jspService(__x1434944766_0_0.java:610)
            at weblogic.servlet.jsp.JspBase.service(JspBase.java:35)
            at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
            at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
            Truncated. see log file for complete stacktrace
    >

This is line from second log:

<<ERROR>> [Feb 29 12:51:36] [[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] [CM] <PayChannelServicesBean.l3UpdatePcnBalanceExpirationIndicator> encountered an exception. This bean uses Container-Managed transactions. Hence, this flow will NOT be retried. Last retry count = <0 out of 0>

Thanks
Sharon


(Al) #4

Just on a side note, if your looking to test your multi-line patterns, you can use the filebeat-multiline-tester tool. I haven't had much feedback on it yet, although it's been useful for me a bunch of times to test various patterns for filebeat configs.


(Christian Dahlqvist) #5

If the patterns are always in separate files, can you not just simplify it by defining multiple prospectors?


(Sharon Sasporta) #6

yes, sure, I define two different prospector and I define multiline.pattern per each prospector.

The issue is to define the right multiline.pattern.

I was thinking about:

multiline.pattern: '^[[:graph:]][[:alpha:]][[:space:]][[:digit:]]{2}[[:graph:]][[:space:]][[:digit:]]{4}[[:space:]][[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}'

and

multiline.pattern: '^[[:graph:]][[:alpha:]][[:graph:]]'
       multiline.negate: true

accordingly


(Sharon Sasporta) #7

It is not clear to me how to install and run the tester.

I download and put the directory on my unix.

What now?

Thanks
Sharon.


(Sharon Sasporta) #8

still waiting for help here, how to define the pattern. Is what I did, is on the right direction? Should it be simpler?


(Christian Dahlqvist) #9

Do the files only contain events with these patterns or are there also events that are not multiline? It would help if you could provide a sample from each file type, e.g in a gist.


(Sharon Sasporta) #10

Wow. Great. I will load some sample to the my github and send you the link.

Thanks,
Sharon.


(Sharon Sasporta) #11

Here are the log files examples:

https://gist.github.com/ssasporta/8776da18c72fbd23f907154e3bb83b54.js


(Christian Dahlqvist) #12

Something looks wrong with this Gist. Can you try recreating it with just text files?


(Sharon Sasporta) #13

(Christian Dahlqvist) #14

For the CMServer.log file it looks to me like every new event begins with a line starting with <<, so you should be able to build a pattern around this if that is the case.

For the ABPServer.log file it looks like every new event begins with a line starting with ####, so you should be able to build a pattern around this if that is the case.

The weblogic.log file seems more complicated and I am not sure how you would like the multiline events to be assembled.

I do unfortunately not have time to write and test this myself at the moment, but hopefully these pointers will get you started.


(Steffen Siering) #15

Still having problems opening the gist files.

Being lazy I just copied all content shown in this discussion and used this regex ^[#\<]

The trick about multiline is not looking at the content, but looking at the structure and re-occuring patterns at beginning or end of lines.

We do have a playground script for users to test multiline patterns. I adapted the script to include all your logs: https://play.golang.org/p/3Eneqg-oN5

Also check out the multiline tester: https://github.com/hartfordfive/filebeat-multiline-tester .
Find executables here: Releases


(Sharon Sasporta) #16

great, as I have more logs and I will have to use this playground/tester.

Anyway, are you saying that the regex ^[#\<] fits all my various format? Should I put it in all the prospectors?

Currently the filebeat is failing to start. A log isn't even created in /var/log/filebeat. I am trying to understand what is wrong there.

If you have any direction for me, it will be perfect.

Thanks
Sharon.


(ruflin) #17

For the starting: Can you try to run it with -e -d "*" and see if you get some output to the console?


(Christian Dahlqvist) #18

I do not think it will work for your weblogic.log file, as you in that file seem to have lines that I suspect should be part of a multiline entry that begin with <. Maybe something like ^[\<][A-Z][a-z]{2}[[:space:]], which matches the first part of the timestamp, could work for this file, although I have not tested it.


(Sharon Sasporta) #19

Thanks. I will test it and let you know.


(Steffen Siering) #20

You are right. The weblogic one looks quite crazy. I'd definitely put weblogic.log into a separate prospector. weblogic.log always starts with <, plus date. You can try to be a little more strict by doing ^\<[JFMASOND][a-z]{2} \d{2}, or try something like ^\<[^\<]. The latter pattern matches a string starting with < not followed by another <. The pattern [^...] negates the characters listed.