Multiline / Aggregate Filter Plugin - Mixed & Muddled Log File


(Lloyd Armstrong) #1

Hi All,

TL;DR

Can the Multiline filter plugin handle a log file that has multi-line events occurring concurrently and interweavingly ?

Explanation

I have a complex query about multi-line logs that are interwoven with other multi-line logs. Here is a snippet of what this looks like. Note: In the below snippet, Thread 6 and Thread 7 are running simultaneously

Each time an event occurs, it is logged from start to finish over multiple log lines. However, other events can be happening simultaneously on a different thread. Therefore, the logs get mixed together. Can the Aggregate filter plugin handle interwoven logs of this type? Any feedback will be greatly appreciated.

THANK YOU!


(Дмитрий Сибильков) #2

Hi! Have the same problem. My log looks like:

[29-Oct-2018 15:19:41] WARNING: [pool www] child 28 said into stdout: "{start very big json line"
[29-Oct-2018 15:19:41] WARNING: [pool www] child 28 said into stdout: "CONTINUE_MARK continue very big json line"
[29-Oct-2018 15:19:41] WARNING: [pool www] child 256 said into stdout: "{unexpected json from another worker}"
[29-Oct-2018 15:19:41] WARNING: [pool www] child 28 said into stdout: "{CONTINUE_MARK end of very big json line}"

It is php-fpm log, which shows workers events. I can output it in simple format (not in JSON), but how can I aggregate these lines?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.