Combine 2 events into one

[bishaka (bowow)]

Hi,
I am trying to merge 2 log events into a single event in logstash..

After some research, i found out about the aggregate filter.. I am not quite sure if I am doing this correctly but as of now.. It is not adding them into a single event..

The event logs I am trying to combine are:

11:31:04,675 INFO [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' device:50 - imsi : 324234324
11:31:04,797 INFO [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' device:70 -imsi : 324234324 completed in 122 ms

As you can see..the events are identified as a single event through the unique IMSI..

A snippet of the config file

if [Device_ID] {
mutate {
add_field => { "State" => "Processing" }
}
} else {
mutate {
add_field => { "State" => "Completed" }
}
}

    if [State] == "Processing" {
            aggregate {
                    task_id => "%{IMSI}"
                    code => "map['sql_duration'] = 0"
                    map_action => "create"
            }
    } else if [State] == "Completed" {
            aggregate {
                    task_id => "%{IMSI}"
                    code => "map['sql_duration'] += event['Response_Time']"
                    end_of_task => true
                    timeout => 120
                    map_action => "update"
            }
    }

In the output..I am seeing this..without any aggregate failure message

"message" => "11:31:04,797 INFO [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' device:70 -imsi : 324234324 completed in 122 ms",
"@version" => "1",
"@timestamp" => <>
"path" => <>
"type" => "<>",
"IMSI" => "<>",
"Response_Time" => 42,
"State" => "Completed"
}
{
"message" => "11:31:04,675 INFO [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' device:50 - imsi : 324234324",
"@version" => "1",
"@timestamp" => <>
"path" => <>
"host" => "<>",
"type" => "<>",
"IMSI" => <>,
"State" => "Processing"

Can someone please with this?

[iti]

Try this:
filters{
aggregate {
task_id => "%{IMSI}"
code => "map['sql_duration'] = event['Response_Time']
map['yourfield1'] ||= event.get('yourfield1')
map['yourfield2'] ||= event.get('yourfield2')
... all fields you need copy from event to map, which will create a new combine event"
push_previous_map_as_event => true
timeout => 3
timeout_tags => ['aggregated']
}
}

output{
if "aggregated" in [tags] { #this will output only the merged event, so is your processing is here
stdout {
codec => rubydebug
}
}
}