I've been trying to combine log lines based on a thread number, but the problem is that multiple processes write to a single log file, meaning we have process blocks interlaced with other process blocks. I want to be able to combine them into multiline events based on thread number, parse them and ship them to elasticsearch
Here's a simplified example of our log lines:
2017-08-19 14:00:01,851 DEBUG thread-19 Method begin
2017-08-19 14:00:01,852 DEBUG thread-19 Log lines
2017-08-19 14:00:01,853 DEBUG thread-19 Log lines
2017-08-19 14:00:01,854 DEBUG thread-20Method begin
2017-08-19 14:00:01,855 DEBUG thread-20 Log lines
2017-08-19 14:00:01,856 DEBUG thread-19 Method End
2017-08-19 14:00:01,857 DEBUG thread-20 Method End
Ideally I'd like to have them grouped like this:
2017-08-19 14:00:01,851 DEBUG thread-19 Method begin
2017-08-19 14:00:01,852 DEBUG thread-19 Log lines
2017-08-19 14:00:01,853 DEBUG thread-19 Log lines
2017-08-19 14:00:01,856 DEBUG thread-19 Method End
2017-08-19 14:00:01,854 DEBUG thread-20Method begin
2017-08-19 14:00:01,855 DEBUG thread-20 Log lines
2017-08-19 14:00:01,857 DEBUG thread-20 Method End
However I'm not sure how to go about doing this. I thought stream_identity would be a good fit but the multiline filter was deprecated some time ago. Is there a way to do this or is it a pointless endeavor? Let me know if there's anything I haven't explained properly.
Many thanks!