Logstash 7.6.0 regression

[toog]

With the same filter, logstash 7.6 have not the same result than previous version of logstash 6.5.4.
I have include on the filter part some ruby counter and for one log in the 6.5.4 version all the counters have the same value for each event.
On the 7.6.0 version the counters goes to increase for same event => is not normal.

the two version are launch with the same command line :
./bin/logstash -r -f ./config/filter.conf -w 1

Result for an event in 6.5.4 version :

{
    "Count_lineBase_ruby": 1,
    "Count_lineBase2_ruby": 1,
    "Count_lineBase3_ruby": 1,
    "Count_lineBase4_ruby": 1,
    "Count_lineBase5_ruby": 1,
    "Count_lineBase6_ruby": 1,
    "Count_lineBase7_ruby": 1,
    "@timestamp": "2020-11-20T07:34:42.500Z",
    "raw_log": "Begin",
    "@version": "1",
    "tags": [
        "MATCH",
        "DATE_FORMAT_OK",
        "BEGIN",
        "SUCCESS"
    ],
    "message": "Nov 20 08:34:42,500 Begin",
    "timestamp": "Nov 20 08:34:42,500",
    "taskid": "data"
}

Result for an event in 7.6.0 version :

{
    "Count_lineBase_ruby": 1,
    "Count_lineBase2_ruby": 1,
    "Count_lineBase3_ruby": 1,
    "Count_lineBase4_ruby": 1,
    "Count_lineBase5_ruby": 1,
    "Count_lineBase6_ruby": 5,
    "Count_lineBase7_ruby": 5,
    "message": "Nov 20 08:34:42,500 Begin",
    "taskid": "data",
    "@timestamp": "2020-11-20T07:34:42.500Z",
    "@version": "1",
    "timestamp": "Nov 20 08:34:42,500",
    "tags": [
        "MATCH",
        "DATE_FORMAT_OK",
        "BEGIN",
        "SUCCESS"
    ],
    "raw_log": "Begin"
}

logs :

Nov 20 08:34:42,500 Begin
Nov 20 08:34:42,501 Data : 1,1,77,1,2,2,3 1,1,2,1,2,2,3
Nov 20 08:34:42,503 End
Nov 20 08:35:11,504 other logs
Nov 20 08:34:42,505 Data : 77,1,2,16,1666,3,2 77,2,2,16,1666,3,2

filter :

filter 
{
ruby {
    init => "@count = 0"
    code => "
    @count=@count+1
    event.set('Count_lineBase_ruby', @count)
"
}

# Get log headers and raw part #
grok {
    remove_tag => ["_grokparsefailure"]
    match => ["message", "%{MONTH:month}%{SPACE}%{MONTHDAY:day} %{TIME:time} %{GREEDYDATA:raw_log}"]
    add_tag => ["MATCH"]
    add_field => {
        "timestamp" => "%{month} %{day} %{time}"
    }

    remove_field => ["month", "day", "time"]
}

ruby {
    init => "@count = 0"
    code => "
    @count=@count+1
    event.set('Count_lineBase2_ruby', @count)
"
}

date {
    match => ["timestamp", "MMM d HH:mm:ss,SSS", "MMM d HH:mm:ss,SSSSSS", "MMM d HH:mm:ss.SSS"]
    timezone => "Europe/Paris"
    add_tag => ["DATE_FORMAT_OK"]
}

ruby {
    init => "@count = 0"
    code => "
    @count=@count+1
    event.set('Count_lineBase3_ruby', @count)
"
}

if "MATCH" in [tags] {
    ruby {
        init => "@count = 0"
        code => "
        @count=@count+1
        event.set('Count_lineBase4_ruby', @count)
    "
    }

    ##############################################
    # GEt all dump of data beetwin Begin and End #
    ##############################################

    # Get Begin aggregation log
    if "SUCCESS" not in [tags] {
        grok {
            remove_tag => ["_grokparsefailure"]
            match => ["raw_log", "^Begin"]
            add_tag => ["BEGIN", "SUCCESS"]
            add_field => {
                "taskid" => "data"
            }
        }
    }

    ruby {
        init => "@count = 0"
        code => "
        @count=@count+1
        event.set('Count_lineBase5_ruby', @count)
    "
    }

    # Get Data log
    if "SUCCESS" not in [tags] {
        grok {
            remove_tag => ["_grokparsefailure"]
            match => ["raw_log", "^Data : %{GREEDYDATA:dump_raw}"]
            add_tag => ["DATA", "SUCCESS"]
            add_field => {
                "taskid" => "data"
            }
        }
    }

    ruby {
        init => "@count = 0"
        code => "
        @count=@count+1
        event.set('Count_lineBase6_ruby', @count)
    "
    }

    # Get End log
    if "SUCCESS" not in [tags] {
        grok {
            remove_tag => ["_grokparsefailure"]
            match => ["raw_log", "^End"]
            add_tag => ["END", "SUCCESS"]
            add_field => {
                "taskid" => "data"
            }
        }
    }

    ruby {
        init => "@count = 0"
        code => "
        @count=@count+1
        event.set('Count_lineBase7_ruby', @count)
    "
    }
}
}

[Badger]

Does adding '--java_execution false' restore the old behaviour?

[toog]

Yes it's better with

--java-execution false

So we can't new java engine ?

I made this example of filter because initialy i discover an execution bug with the use of the plugin aggregation and to understand why logstash mixed up my logs.

[Badger]

You are making assumptions about the order in which events are processed by filters. Those assumptions are not currently valid.