Organising multiline events with multithreaded log lines


(Andrew McBride) #1

I've been trying to combine log lines based on a thread number, but the problem is that multiple processes write to a single log file, meaning we have process blocks interlaced with other process blocks. I want to be able to combine them into multiline events based on thread number, parse them and ship them to elasticsearch
Here's a simplified example of our log lines:

2017-08-19 14:00:01,851 DEBUG thread-19 Method begin
2017-08-19 14:00:01,852 DEBUG thread-19 Log lines
2017-08-19 14:00:01,853 DEBUG thread-19 Log lines
2017-08-19 14:00:01,854 DEBUG thread-20Method begin
2017-08-19 14:00:01,855 DEBUG thread-20 Log lines
2017-08-19 14:00:01,856 DEBUG thread-19 Method End
2017-08-19 14:00:01,857 DEBUG thread-20 Method End

Ideally I'd like to have them grouped like this:

2017-08-19 14:00:01,851 DEBUG thread-19 Method begin
2017-08-19 14:00:01,852 DEBUG thread-19 Log lines
2017-08-19 14:00:01,853 DEBUG thread-19 Log lines
2017-08-19 14:00:01,856 DEBUG thread-19 Method End
2017-08-19 14:00:01,854 DEBUG thread-20Method begin
2017-08-19 14:00:01,855 DEBUG thread-20 Log lines
2017-08-19 14:00:01,857 DEBUG thread-20 Method End

However I'm not sure how to go about doing this. I thought stream_identity would be a good fit but the multiline filter was deprecated some time ago. Is there a way to do this or is it a pointless endeavor? Let me know if there's anything I haven't explained properly.
Many thanks!


(Jake Landis) #2

You may want to take a look at https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.