Multiline / Aggregate Filter Plugin - Mixed & Muddled Log File

Elastic Stack Logstash
1

Hi All,

TL;DR

Can the Multiline filter plugin handle a log file that has multi-line events occurring concurrently and interweavingly ?

Explanation

I have a complex query about multi-line logs that are interwoven with other multi-line logs. Here is a snippet of what this looks like. Note: In the below snippet, Thread 6 and Thread 7 are running simultaneously

Logs
Logs.PNG935×886 79 KB

Each time an event occurs, it is logged from start to finish over multiple log lines. However, other events can be happening simultaneously on a different thread. Therefore, the logs get mixed together. Can the Aggregate filter plugin handle interwoven logs of this type? Any feedback will be greatly appreciated.

THANK YOU!

2

Hi! Have the same problem. My log looks like:

[29-Oct-2018 15:19:41] WARNING: [pool www] child 28 said into stdout: "{start very big json line"
[29-Oct-2018 15:19:41] WARNING: [pool www] child 28 said into stdout: "CONTINUE_MARK continue very big json line"
[29-Oct-2018 15:19:41] WARNING: [pool www] child 256 said into stdout: "{unexpected json from another worker}"
[29-Oct-2018 15:19:41] WARNING: [pool www] child 28 said into stdout: "{CONTINUE_MARK end of very big json line}"

It is php-fpm log, which shows workers events. I can output it in simple format (not in JSON), but how can I aggregate these lines?