Specific GROK filter for multi-line Postgresql log

Elastic Stack Logstash
9

@111126

Ok, given your logstash configuration, I understand now your need.
Your main problem is that you have no explicit "end line marker" to flush aggregate map.
But you're lucky ! Last week, aggregate plugin has been released with new options to deal with that case !

So, here's the right configuration for your need :

    grok {
        match => [ "message", "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:logsource} %{SYSLOGPROG}: \[%{INT:line}-%{INT:part_of_line}\] %{GREEDYDATA:ostatok}" ]
    }
    
    aggregate {
        task_id => "%{line}"
        code =>  "
            map.merge!(event) if map.empty?
            map['full_message'] ||= ''
            map['full_message'] += event['ostatok']
        "
        timeout => 10
        push_map_as_event_on_timeout => true
        timeout_code => "event.tag('aggregated')"
    }

    if "aggregated" not in [tags] {
        drop {}
    }

Note that in your grok expression, [ and ] chars must be escaped.
And for aggregate, the main idea is that, as you have no explicit "end log line", we use 10s timeout to push aggregated map as a new logstash event in the pipeline.

1 Like