Hi everyone,
Any help would be greatly appreciated.
We recently upgraded our kubernetes cluster and all logs are now coming in with the new kubernetes CRI log format, that includes a timestamp and the output stream type prepending every log event example:
2023-10-06T00:17:09.669794202Z stdout F Exception in thread "main" java.lang.NullPointerException
2023-10-06T00:17:09.669794202Z stdout F at com.example.myproject.Book.getTitle(Book.java:16)
2023-10-06T00:17:09.669794202Z stdout F at com.example.myproject.Author.getBookTitles(Author.java:25)
2023-10-06T00:17:10.113242941Z stdout F at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
I am able to sucessfully remove the new CRI logs by using the dissect filter in Filebeat on my filestream input, to extract the log messages before sending it to Logstash.
However, I'm running into issues with multiline log events. From my testing, I haven't been able to first extract the log messages using dissect, then apply the multiline parser after to aggregate log events. Multiline never seems to go through as I am assuming it is being applied to log events before it's been dissected.
Does anyone know if running multiline after dissect is possible? Thanks in advance for your time and any help