Aggregate messages with different patterns


An application we manage generates logs with different patterns.

One event generates between 2 and 4 messages, which have the same event_id in the lig message (in the below example: "event_id_1"

Each message with the same event_id has a code_number and a sub_code_number. Depending of the couple code_number/sub_code_number, the message pattern is different.

For example, I have the following log messages associated to "event_id_1"
log1: event_id_1 code_number_1 sub_code_number_1
log2: event_id_1 code_number_1 sub_code_number_2 log

I know the messages pattern for code_number_1 events, so I know where to find information.
I want to extract:

  • info1 inside first message
  • info 4 inside second message

(in case of code_number_2 event, message patterns would be different, ans I'd need to extract another information in another place inside the messages).

I am totally beginner and I am not sure about how to process with that. What I expacted to do is something like that:
Step #1: extract "event_id", code_number" and "sub_code_number" with grok
Step #2: use "aggregate" filter to analyze all message with the same event_id (using a timeout as there is non start/end information in the message)
Step#3: inside the aggregate, use another time "grok" to extract the fields which I want?

Thanks for your help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.