Logstash Filter - Grok pattern not working as expected

[pavank (Pavana Kumar N)]

Hi All,

We are trying to apply Grok filter patterns for our application logs. The logs are applied successfully on the grok debugger online as well as Kibana Dev Tool debugger, but is not getting applied properly in the Logstash.

Log sample:
[12/21/20 9:07:30:884 UTC] 00000001 WsServerImpl E WSVR0009E: Error occurred during startup
com.ibm.ws.exception.RuntimeError: com.ibm.ejs.EJSException: Could not register with Location Service Daemon, which could only reside in the NodeAgent. Make sure the NodeAgent for this node is up and running.; nested exception is:
com.ibm.ejs.EJSException: Could not register with Location Service Daemon, which could only reside in the NodeAgent. Make sure the NodeAgent for this node is up and running.; nested exception is:
java.lang.NullPointerException
at com.ibm.ws.runtime.component.ORBImpl.start(ORBImpl.java:486)
at com.ibm.ws.runtime.component.ContainerHelper.startComponents(ContainerHelper.java:540)
at com.ibm.ws.runtime.component.ContainerImpl.startComponents(ContainerImpl.java:627)
at com.ibm.ws.runtime.component.ContainerImpl.start(ContainerImpl.java:618)
at com.ibm.ws.runtime.component.ServerImpl.start(ServerImpl.java:555)
at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:311)
at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:224)
at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:697)
at com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Grok pattern:
[%{GREEDYDATA}]%{SPACE}%{WORD}%{SPACE}%{WORD:ShortName}%{SPACE}%{WORD:Loglevel}%{SPACE}%{DATA:ErrorCode}:%{SPACE}%{GREEDYDATA:Description}\n%{GREEDYDATA}\n%{SPACE}%{GREEDYDATA}(?m)\n%{SPACE}%{GREEDYDATA}

I need the bold parts of the logs to be captured under the fields respectively, but under Description (also attached screenshot) , I see that the entire log lines being captured when I check in Kibana.
Any help pointing to any mistakes would be of great help.

Regards,
Pavan

[Badger]

@pavank

%{GREEDYDATA:Description}\n%{GREEDYDATA}\n%{SPACE}%{GREEDYDATA}(?m)\n%{SPACE}%{GREEDYDATA}

Your pattern has three newlines in it. In a grok filter, the [Description] field, because it is greedy, will capture as much as possible. Probably everything except the last two lines of the message, because they are used to match the other, unnamed fields.

You could change %{GREEDYDATA:Description}\n to %{DATA:Description}\n, which would capture the minumum required to match, which would be the rest of the line. Alternatively (and I think the intent is clearer) capture everything that is not a newline, up to the next newline (?<Description>[^\n]+)\n.

[pavank (Pavana Kumar N)]

Hello Badger.

This helped. Using DATA for logs with information in a single line nand GREEDYDATA for logs with information spanning multiple lines worked for me.
Thank you very much for the help.
A very Happy New Year to you.

Regards,
Pavan