Multiline pattern filebeat

muralibala · January 3, 2017, 7:14pm

I have the filebeat prospector setup for a custom log file. The errors are in multiple lines.

Ex:
####<Jan 1, 2017 2:31:53 PM EST> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1483299113805> <[ServletContext@371700326[app:TC module:DeltekTC path:null spec-version:null]] Servlet failed with an Exception
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:454)
at java.lang.Integer.parseInt(Integer.java:527)
at jsp_servlet._com._deltek._tc._ts.__timesheet2._jspService(__timesheet2.java:1545)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:221)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3284)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3254)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2163)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2089)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2074)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1513)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:254)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

My filebeat.prospectors looks like this:

input_type: log

Paths that should be crawled and fetched. Glob based paths.

paths:
- C:\logs\DTServer.log

document_type: DTLog

ignore_older: 5h
multiline.pattern: '^[1]'
multiline.negate: false
multiline.match: after

So multiline.pattern: '^[2]' does not work. Multiple empty log entries are being created.

What would be my regex for the multiline pattern to get everything between ####< and >

TIA

[:space:] ↩︎
[:space:] ↩︎

steffens · January 3, 2017, 9:05pm

can you please format you logs and config file using the </> button?

Can you add some more log lines for us to get a better idea about the log files structure? E.g. if a log-entry always starts with <, why not use this one for filtering?

ruflin · January 5, 2017, 12:15pm

The playground here is very useful to test your regexp expressions: https://www.elastic.co/guide/en/beats/filebeat/5.1/multiline-examples.html#_testing_your_regexp_pattern_for_multiline

muralibala · January 5, 2017, 2:44pm

All log enteries are in the following format:

####<Jan 5, 2017 8:26:51 AM EST> <Error> <com.deltek.tc.framework> <Server> <TEServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <1382934[0$X]WESTAT> <> <> <1483622811881> <BEA-000000> <FrontServlet:doPost:
******************** Exception 0 ********************
java.lang.NullPointerException
	at com.deltek.tc.framework.EssAppController.doPost(EssAppController.java:44)
	at com.deltek.tc.framework.Controller.doGet(Controller.java:32)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:221)
	at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:567)
	at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:263)
	at com.deltek.tc.framework.RequestProcessor.doPost(RequestProcessor.java:42)
	at com.deltek.tc.framework.RequestProcessor.doGet(RequestProcessor.java:28)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:221)
	at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:567)
	at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:263)
	at com.deltek.tc.framework.FrontServlet.doPost(FrontServlet.java:292)
	at com.deltek.tc.framework.FrontServlet.doGet(FrontServlet.java:316)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:221)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3284)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3254)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
	at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
	at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2163)
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2089)
	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2074)
	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1513)
	at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:254)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

>

muralibala · January 5, 2017, 2:45pm

So the log enteries always starts with ####< and ends with >

muralibala · January 5, 2017, 2:50pm

Awesome. Will give that a shot.

muralibala · January 5, 2017, 6:42pm

Steffens,

This worked:

  multiline.pattern: '<*'
  multiline.negate: false
  multiline.match: after

system · February 2, 2017, 6:42pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat multiline is not working Beats filebeat	2	746	May 1, 2018
Filebeat not honoring multiline configuration Beats filebeat	3	655	May 25, 2017
Filebeat 6.0 Multiline issue Beats filebeat	17	1509	December 27, 2017
Need help for the correct multline pattern Beats filebeat	2	353	May 7, 2018
I want to run two multiline pattern using filebeat Beats filebeat	1	310	June 10, 2020

Multiline pattern filebeat

Paths that should be crawled and fetched. Glob based paths.

Related topics