Multiline pattern for DateTime not working when logs are comming with high rate

[rudok (Krejci Rudo)]

Hello

I am trying to get tomcat log messages base on date time pattern:

multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3} '
multiline.negate: true
multiline.match: after

When logs was come in slow rate ( 1-10) seconds - the pattern works, but when logs come in range of
millisecond , two logs was merged as one log.

Logs example:
2018-03-04 08:53:55.496 ERROR {ContainerBackgroundProcessor[StandardEngine[Catalina]]} o.a.c.c.C.[.[localhost].[/blabla##01] [StandardSession.java : 835] : Session event listener threw exception
java.lang.NullPointerException: null
at com.ulcjava.container.servlet.server.ServletContainerAdapterHelper.getIds(ServletContainerAdapterHelper.java:319)
at com.ulcjava.container.servlet.server.HttpSessionStore.getSessionIds(HttpSessionStore.java:23)
at com.ulcjava.container.servlet.server.HttpSessionListener.sessionDestroyed(HttpSessionListener.java:2)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:824)
at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:682)
at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:552)
at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:537)
at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5554)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1388)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1392)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1392)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1360)
at java.lang.Thread.run(Thread.java:745)
2018-03-04 08:53:55.497 ERROR {ContainerBackgroundProcessor[StandardEngine[Catalina]]} o.a.c.c.C.[.[localhost].[/blabla##01] [StandardSession.java : 835] : Session event listener threw exception
java.lang.NullPointerException: null
at com.ulcjava.container.servlet.server.ServletContainerAdapterHelper.getIds(ServletContainerAdapterHelper.java:319)
at com.ulcjava.container.servlet.server.HttpSessionStore.getSessionIds(HttpSessionStore.java:23)
at com.ulcjava.container.servlet.server.HttpSessionListener.sessionDestroyed(HttpSessionListener.java:2)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:824)
at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:682)
at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:552)
at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:537)
at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5554)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1388)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1392)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1392)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1360)
at java.lang.Thread.run(Thread.java:745)

Is there any way to solve that?

Thx

Rudo

[ruflin]

I'm not sure why filebeat multiline would merge the two into 1 event. Can you try it out with file output enabled and see how many events you get with the above?