Logstash - aggregate filter - for each transaction/thread, only look for the first match message

[Vincent_Chen]

Hi everyone,

We have a log that do not provide a unique ID for each transaction.

We want to use the thread-id and the message body to define the transaction boundary.

Here is the mock-up log:

##transaction 1
[2019-09-27 10:16:50,170], info, thread1, start
[2019-09-27 10:16:50,171], info, thread1, received
[2019-09-27 10:16:50,172], info, thread1, end
#transaction 2
[2019-09-27 10:16:51,180], info, thread1, start
[2019-09-27 10:16:51,181], info, thread1, received
[2019-09-27 10:16:51,182], info, thread1, end

Here is my filter:

  if [msg] =~ "start" {
aggregate {
  task_id => "%{thread}"
  code => "map['new_msg'] ||= event.get('message')"
  map_action => "create"
  }
  }

  if [msg] =~ "received" {
  aggregate {
  task_id => "%{thread}"
  code => "map['new_msg'] << event.get('message')"
  map_action => "update"
}  
  }

 if [msg] =~ "end" {
aggregate {
  task_id => "%{thread}"
  code => "map['new_msg'] << event.get('message'); event.set('new_msg', map['new_msg'])"
  map_action => "update"
  end_of_task => true
  timeout => 10
}
}  

Here is the output:

[2019-09-27 10:16:50,170], info, threadid-1, start
[2019-09-27 10:16:50,171], info, threadid-1, received
[2019-09-27 10:16:51,181], info, threadid-1, received
[2019-09-27 10:16:50,182], info, threadid-1, end

We can see the process treat the "received" event from the second transaction as the first transaction.

Could we tell the aggregate filter to only look for the first match event?

Cheers,
Vincent