Aggregate filter: How to delete the event when no matching task_id found within the timeout?

Elastic Stack Logstash
1

Hi,

I am using aggregate filter in Logstash to merge the events based on task_id. The problem is, I don't want to push the event when no matching task_id found within the given timeout.

Please pour you ideas to achieve it.

2

What does your aggregate configuration look like?

3 
aggregate {
   task_id => "%{[kvData][flowId]}"
        code => '
            map["kvData"] ||= {}
            map["kvData"] =  map["kvData"].merge(event.get("kvData"))
            map["Type"] = event.get("Type")
            map["Timestamp"] = event.get("timestamp")
                        map["@timestamp"] = event.get("@timestamp")
            event.cancel
        '
         push_map_as_event_on_timeout => true
         timeout_timestamp_field => "@timestamp"
         inactivity_timeout => 10
         timeout => 10
         timeout_tags => ['_aggregatetimeout']
         timeout_task_id_field => "[kvData][flowId]"
}
4

It sounds like you want to drop {} the event if there have not been multiple events combined. If so you could add

 map["eventCount"] ||= 0
 map["eventCount"] += 1

to the code option, and then

if [eventCount] == 1 {
    drop {}
else {
    mutate { remove_field => [ "eventCount" ] }
}

If I have misunderstood your question please explain further.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.