Processing multiline events from filebeat

Elastic Stack Logstash
1

Filebeat has been configured to read file and produce multiline events that are sent to logstash.

In logstash, I want to combine multiple multiline events using grok and aggregate filters.

As part of grok, I also want the original message to be sent to ES.

time some random text key value
Key1 val1
Key2 val2
...

I want to aggregate based on key2.

2

Can you post some sample log lines.
I think you can use Logstash Multline Codec for your purpose.

Edit: And what I would suggest is put your multiline codec at Filebeat itself.

3

I have put the multiline processing in filebeat and it's working. At logstash, I want to aggregate multiple multiline events if the events contain same value for key2. I have pasted a sample message above.

4

If I understand correctly you are trying to group multiple log lines at Logstash also, based on some value which matches in the previous log event. Is that correct?

Edit: If that is the case am not sure how to do it. I am afraid it is not even possible I think.!!

5

12:12:12 processed T1
data1 val1
data2 val2
dataId 1111
data7 val7

Above log lines get combined by beat and sent to logstash as 1 event. I will receive multiple like this that need to be aggregated based on dataId key.