Help with aggregation code

Elastic Stack Logstash
1

We process mails through multiple modules resulting in logs with the same mail_id, kinda like this:
mail_id_X module1: key1 key2
mail_id_X module2: key3 key4
mail_id_X module3: key5 key6 key7

What I would like to do is take key5 and key6 from module3 and add it to the document from module2, while leaving module3 as it is:
mail_id_X module1: key1 key2
mail_id_X module2: key3 key4 key5 key6
mail_id_X module3: key5 key6 key7

As module3 can vary or be the same as module1, I understand I have to use push_map_as_event_on_timeout, but given my lack of Ruby knowledge, I am complete clueless on the code I need for the aggregation.

filter {
         grok {
          match => {
            "message" =>  "{WORD:modul}{GREEDYDATA:log_message}" 
          }
        }
        
        if "modul1" in [modul] {
            grok..
        }
        if "modul2" in [modul] {
            grok..
        }
        if "modul3" in [modul] {
            grok..
        }

        aggregate {
             task_id => "%{mail_id}"
             push_map_as_event_on_timeout => true
             code => ""
             timeout_task_id_field => "mail_id"
             timeout => 5
             timeout_tags => ['_aggregatetimeout']
             }
}

Any help would be appreciated!

2

Hard to answer without knowing what the module2 and module3 events look like. Which fields do you want to combine from each one?

3

Hm sorry, I thought the example lines are sufficient.
module2 is a spam check and module3 includes the mail subject and a route number, which I'd like to combine with the spam check result. So one can just browse the spam check logs and instantly see the subject and route too.

4

That really doesn't tell me what the events look like, but I would guess at something like

code => '
    map["@timestamp"] ||= event.get("@timestamp")
    map["spamCheck"] ||= event.get("spamCheck")
    map["subject"] ||= event.get("subject")
    map["routeNumber"] ||= event.get("routeNumber")
'
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.