Problem with aggregation filter


#1

Hi,
I'm new to the aggregation filter and would really appreciate some assistance in working with it to solve the following problem.

My log data contains certain lines which have a data value which must be persisted, or remembered and added as field to subsequent events.

Lets say, for example that my log line looks like:

init Starting run.
init param b set to "jellybeans"
init startup state: begin
init startup state: complete
init state: running
init param b set to "gum"
init startup state: begin
init startup state: complete
init state: running

I am grokking "jellybeans" into a field, lets call it: "object". That's simple.

But getting the "object" field into the subsequent events is not. From reading about it, it sounds like aggregate should be able to do this.

I need the startup line events to also contain either "jellybeans" or "gum" depending on whichever is the "newest".
This is needed to group the events by the param b value in Elastic/Kibana.

I have:

if [message] =~ /param\sb\sset\sto/ {
    grok {			
        match => { "message" => ".*set\sto\s\"(?<object>\w*)" }
    }
    mutate {
        add_field => { "utility_field" => "all" }
    }
    aggregate {
        task_id => "%{utility_field}"
        map_action => "create_or_update"
        code => "map['utility_field'] = event['object']"
    }
}

with that I had expected the utlity_field to now contain "jellybeans". But it does not.

I have tried adding the aggregate code to the other events but the expected field (utility_field) never gets updated with the value of the "object" field.

I have tried different combinations of the use of field too, but nothing seems to influence the content of the "utility_field" - it only ever contains the "all" string.

My understanding is that the aggregate filter should be set to "create" or "update" on events which contain something you are interested in (in my case "object" strings ). But it has to operate on a valid and existing field (utility_field).. by mapping the value of that field to the data you care about from the current event.

Is that correct? Where am I going wrong?

Many thanks.


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.