Trying to make logstash work with multiline logs on a SIEM platform

[currybread (Bryan Chan)]

Situation: Multiline logs collected are sent to a SIEM event collector via Logstash but having issues

test.log:

[5/8/22 7:31:23:546 SGT]     FFDC Exception:java.io.FileNotFoundException 
SourceId:com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters -IOE ProbeId:1044
java.io.FileNotFoundException: CEAR1251G: File not found: /favicon.ico
	at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)
	at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:958)
	at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:486)
	at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
	at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4075)
	at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
	at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1019)
	at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
	at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:213)
	at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
	at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
	at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
	at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
	at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
	at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
	at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
	at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
	at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
	at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
	at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
	at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
	at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
	at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
	at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Reporter:null`Preformatted text`

When the logs are received at SIEM event collector, seems it stopped at timestamp and the rest of the logs are ignored

<13>1 - - - - - [lc@36807 lc.ctime="1659078412480" lc.cid="117-Log" lc.ctype="logstash"] {"host":"117-Log","path":"/var/log/test.log","@version":"1","tags":["_grokparsefailure"],"message":"\r","@timestamp":"2022-07-29T07:06:52.480Z"}

Input and filter configuration

input {
file {
path => "/var/log/test.log"
start_position => "beginning"
ignore_older => 2
}
}

filter {
grok {
match => [ "message", "(?m)[%{GREEDYDATA:timestamp}]\s%{DATA:logtype}\sException:+%{DATA:exception}\nSourceId:+%{DATA:sourceid}\njava.io+%{DATA:javaio}\n%{DATA:package1}\n%{DATA:package2}\n" ]
}

}

[Rios]

This grok should work:
(?m)[%{DATA:timestamp}]\s+%{WORD:logtype}\sException:%{DATA:exception}\sSourceId:%{DATA:source}\s+ProbeId:%{DATA:probe}\s+CEAR1251G:\s%{GREEDYDATA:log}