Begin collate or multiline with one conditional, but include more once running

(Mark Casey) #1


I'm planning to use the multiline filter or (the actual current plan) the collate filter to combine events into digests for emailing, similar to this example I found when searching around...:

input {

filter {
    grok {
       ... # creates a "http_response" field
    if [http_response] =~ /^5\d\d$/ {
        clone {
            add_tag => "_for_collation"
            clones => "all"
        if "_for_collation" in [tags] {
            collate {
                add_tag => "_collated"
    } else {
        drop { }

output {
    if "_for_collation" in [tags] {
        email {

My question is whether it is possible to "start" a collation with a conditional (such as the '=~ /^5\d\d$/') used above, but once a collation is running use another set of conditionals to include more events such as "all apache-access that isn't http200OK" plus "all application-log where message is not 'access denied please login'".

I can code up that list of conditionals and I already have some notifications working via email... I'm just stumped on the "if $collation_already_running" part that it seems should wrap them.

In case it is relevant the pipeline is LSF->LS->Redis->LS->ES/email. LS is v1.4.2.

Thank you,

(Mark Casey) #2

Alternatively... if it isn't too terrible for the performance... maybe I could run the collation all the time on the wider set of conditionals but have a metrics event come by that drops it if no undesirable events have been counted?

(system) #3