Multiline - multiple level of multiline in Filebeat

[aviman (Avijit Mandal)]

Hi,

Can we use multiple level of multiline feature in Filebeat.

Input --> multiline (to join based on 1st pattern) --> processor(drop lines based on condition) -->multiline (to join based on 2pattern)

Input :
0 STARTED Thu Jan 30 22:48:00 2020
Starting process1
1 INFO Thu Jan 30 22:48:01 2020
process1 variable
2 STARTED Thu Jan 30 22:53:35 2020
process1 aborted.

Output (line to be send to Logstash)

1. Multiline ouput
   |0|STARTED|Thu Jan 30 22:48:00 2020\n\tStarting process1|
   |---|---|---|
   |1|INFO|Thu Jan 30 22:48:01 2020\n\tprocess1 variable|
   |2|STARTED|Thu Jan 30 22:53:35 2020\n\tprocess1 aborted.|

2. Processor output (Removing INFO)
   |0|STARTED|Thu Jan 30 22:48:00 2020\n\tStarting process1|
   |---|---|---|
   |2|STARTED|Thu Jan 30 22:53:35 2020\n\tprocess1 aborted.|

Till above I could achieve
Now next wanted above line in multiline and get output - a single line:

0 STARTED Thu Jan 30 22:48:00 2020\n\tStarting process1\n2 STARTED Thu Jan 30 22:53:35 2020\n\tprocess1 aborted.

Thanks & Regards,
Avijit Madal

[mtojek (Marcin Tojek)]

Hi,

could you please format your post using </> ? It's really hard to read log entries.

[aviman (Avijit Mandal)]

----My Log--- (starting values 0,1,2..represent log lines spread into multiple lines)

0 STARTED Thu Jan 30 22:48:00 2020
Starting process1
1 INFO Thu Jan 30 22:48:01 2020
process1 variable
2 STARTED Thu Jan 30 22:53:35 2020
process1 aborted.

Expected output to logstash

0 STARTED Thu Jan 30 22:48:00 2020 Starting process1 2 STARTED Thu Jan 30 22:53:35 2020 process1 aborted.

[aviman (Avijit Mandal)]

Hi Marcin,

Let me know if second post makes it clear.
If not will try to rewrite the issue.

Thanks

[mtojek (Marcin Tojek)]

Please rethink if you really need to build complex processing expressions. It makes it harder to maintain. It's easier sometimes to use a script processors: https://www.elastic.co/guide/en/beats/filebeat/master/processor-script.html