ELPASED Filter - Works differently 6.8.1 vs 7.1 - Possible Bug?

Elastic Stack Logstash
4

I cannot think of a way to do it if order is not preserved. In fact the aggregate filter is no longer of much use if order is not preserved even when '-w 1'.

Here is a minimal example that shows the problem. The file /tmp/foo.txt
contains

2019.04.25|09:36:49.850||B8CA3A947330-5CC1B251-1||B8CA3A947330-5CC1B251-146|(null)|MainThread|Generic|Begin|check_patient_for_completeness()|(null)|
2019.04.25|09:36:50.128||B8CA3A947330-5CC1B251-1||B8CA3A947330-5CC1B251-146|(null)|MainThread|Generic|End|check_patient_for_completeness()|(null)|

With 7.1.1 and 1.8.0_201 the following configuration will consistently output the Begin line after the End line (and in the original configuration, which used an elapsed filter, it was clear they were processed through the pipeline in that order).

input { file { path => "/tmp/foo.txt" sincedb_path => "/dev/null" start_position => beginning } }
filter {
    grok {
        pattern_definitions => { "SESSIONUID" => "[-A-Z0-9]+" }
        match => { "message" => "%{YEAR:log_year}\.%{MONTHNUM:log_month}\.%{MONTHDAY:log_day}\|%{TIME:log_time}\|\|%{SESSIONUID:SESUID}\|\|%{SESSIONUID:THREADUID}\|\(null\)\|%{WORD:ThreadName}\|%{WORD:Action}\|Begin\|check_patient_for_completeness%{GREEDYDATA}"}
    }
    if "_grokparsefailure" in [tags]{
        grok {
            pattern_definitions => { "SESSIONUID" => "[-A-Z0-9]+" }
            match => { "message" => "%{YEAR:log_year}\.%{MONTHNUM:log_month}\.%{MONTHDAY:log_day}\|%{TIME:log_time}\|\|%{SESSIONUID:SESUID}\|\|%{SESSIONUID:THREADUID}\|\(null\)\|%{WORD:ThreadName}\|%{WORD:Action}\|End\|check_patient_for_completeness%{GREEDYDATA}"}
        }
    }
}
output { stdout { codec => rubydebug { metadata => false } } }

Very slight changes to this, like simplifying the grok pattern, or commenting out the if so that the second grok is unconditional will reverse the order of the events. I don't see how this can not be a bug, but I don't get to decide that.

1 Like