Event aggregation with logstash error due to mixed task_id

[lusynda]

Hi all
I have a case like this:
my email log has a task_id i called it event.id due to how the log was configed now all the log are seperated and only joined by the event.id field.
Now i've tried to use the aggregate in logstash to put all the log together but then the thing i notices is that event if i set the timeout for the aggregate plugin. It can only mapp all event that has the same event.id that goes after the same event.id

To better explain it i have a little example:
i have 2 id: aaa and bbb
now aaa id have 3 log and bbb have 2
the course of the log will goes like this to elastic:
aaa
aaa
bbb
bbb
aaa

then i notices that the first 2 aaa id will be aggregated. the last aaa id did not get aggregated to the same event as the other 2 since the id bbb got process in the middle.

I want to know if there is any other way to fix this issue to enable me the aggregate all 3 aaa id event in the same event that got push to elastic

[Badger]

What does your aggregate filter configuration look like?

[lusynda]

It looks like this:

filter {
  aggregate {
    task_id => "%{mail.session_id}"
    code => "
      map['@timestamp'] ||= event.get('@timestamp')
      map['source.ip'] ||= event.get('mail.client.ip')
      map['user.name'] ||= event.get('user.name')
      map['mail'] ||= []
      map['mail'] << {'data' => event.get('mail.data')}
      map['mail'] << {'context' => event.get('mail.context')}
    "
    push_previous_map_as_event => true
    timeout_task_id_field => "mail.session_id"
    timeout => 90
    inactivity_timeout => 75
    timeout_tags => ['_aggregatetimeout']
  }
}

[Badger]

@lusynda

push_previous_map_as_event => true

That says to push an event every time the task_id changes, so after seeing two aaa events it sees a bbb event and pushes the two aggregated aaa events. You could try using push_map_as_event_on_timeout instead.