Aggregate not working on certain events (solved)

[Lorenzo_Lapotre]

Hello,
I'm trying to aggregate multiple events that are created upon printing a document, events 842,805,307. However it seems that it is not picking up anything from 842 and 805 events. 842 and 805 carry informations like the color used when printing, username of the person who printed a documents and the number of copies that they printed.

input {
beats{
port => 5044
}
}
filter {

if [event_id] == 842 {
  mutate {
    add_field => {
    "Test" => "Hello"
    }
  }
 aggregate {
   task_id => "%{[user_data][JobId]}"
   code => "map['username'] = event.get('[user][name]')"
 }
}

if [event_id] == 805 {
  mutate {
    add_field => {
    "Test" => "Hello"   
    }
  }
 aggregate {
   task_id => "%{[user_data][JobId]}"
   code => "map['color'] = event.get('[user_data][Color]')
   map['copies'] = event.get('Copies')"
 }
}

if [event_id] == 307 {
  mutate {
    add_field => {
    "Test" => "Hello"
    }
  }

  aggregate {
   task_id => "%{[user_data][Param1]}"
   code => "map['opcode'] = event.get('opcode')"
   push_map_as_event_on_timeout => true
   timeout => 10
  }
}

end result is :

Any help would be greatly appreciated.
Thank you.

[Badger]

Are you sure that [user_data][Param1] in the last filter is equal to [user_data][JobId] in the other filters?

[Lorenzo_Lapotre]

I am not sure. However both [user_data][JobId] and [user_data][Param1] exist. Looking at the values I assumed 'Param1' matched 'JobId'. Because event 307 doesn't have a 'JobId'. Would it be a problem if I'm mistaken and that they don't match?

[Badger]

@Lorenzo_Lapotre

Would it be a problem if I'm mistaken and that they don't match?

Definitely. The aggregate filter groups data in "map" based on the task_id. If the task_id field is different then it will not connect them.

[Lorenzo_Lapotre]

Thanks for the help.
I just checked and they do match.
What else could it be?

[Lorenzo_Lapotre]

I put a 'push_map_as_event_on_timeout => true' for every event and now I get the fields I want, but in different events instead of just one. As I said the JobId and Param1 do match...

[Badger]

Can you add timeout_task_id_field to your filters? I want to make sure they do match.

[Lorenzo_Lapotre]

Problem is solved.
Even though '[user_data][JobId]' and '[user_data][Param1]' did match the aggregate plugin refused to put them together so I added a new field named 'jobid' for 805 and 842 events referencing [user_data][JobId].
Then I used task_id => "%{jobid}".

Thanks for your time and help,
Have a great day.

[Badger]

I was wondering if one was a number, like 1234, and the other was a string, like "1234". They would not match. If you used mutate to copy one field to another then it would be converted to a string if it were a number, which would cause it to start matching.