Processing multiline events from filebeat

Filebeat has been configured to read file and produce multiline events that are sent to logstash.

In logstash, I want to combine multiple multiline events using grok and aggregate filters.

As part of grok, I also want the original message to be sent to ES.

time some random text key value
Key1 val1
Key2 val2

I want to aggregate based on key2.

Can you post some sample log lines.
I think you can use Logstash Multline Codec for your purpose.

Edit: And what I would suggest is put your multiline codec at Filebeat itself.

I have put the multiline processing in filebeat and it's working. At logstash, I want to aggregate multiple multiline events if the events contain same value for key2. I have pasted a sample message above.

If I understand correctly you are trying to group multiple log lines at Logstash also, based on some value which matches in the previous log event. Is that correct?

Edit: If that is the case am not sure how to do it. I am afraid it is not even possible I think.!!

12:12:12 processed T1
data1 val1
data2 val2
dataId 1111
data7 val7

Above log lines get combined by beat and sent to logstash as 1 event. I will receive multiple like this that need to be aggregated based on dataId key.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.