Begin collate or multiline with one conditional, but include more once running


(Mark Casey) #1

Hello,

I'm planning to use the multiline filter or (the actual current plan) the collate filter to combine events into digests for emailing, similar to this example I found when searching around...:

input {
    ....
}

filter {
    grok {
       ... # creates a "http_response" field
    }
    if [http_response] =~ /^5\d\d$/ {
        clone {
            add_tag => "_for_collation"
            clones => "all"
        }
        if "_for_collation" in [tags] {
            collate {
                add_tag => "_collated"
            }
        }
    } else {
        drop { }
    }
}

output {
    if "_for_collation" in [tags] {
        email {
            ...
        }
    }
}

My question is whether it is possible to "start" a collation with a conditional (such as the '=~ /^5\d\d$/') used above, but once a collation is running use another set of conditionals to include more events such as "all apache-access that isn't http200OK" plus "all application-log where message is not 'access denied please login'".

I can code up that list of conditionals and I already have some notifications working via email... I'm just stumped on the "if $collation_already_running" part that it seems should wrap them.

In case it is relevant the pipeline is LSF->LS->Redis->LS->ES/email. LS is v1.4.2.

Thank you,
Mark


(Mark Casey) #2

Alternatively... if it isn't too terrible for the performance... maybe I could run the collation all the time on the wider set of conditionals but have a metrics event come by that drops it if no undesirable events have been counted?


(system) #3