ES grok processor break_on_match => false needed

[ffoysal]

We have a pattern in logstash grok filter like this
grok {
break_on_match => false
match => [ "message",".+mv=(?<mediaVendor>[\w]+)[\s&]+.+",
"message",".+mf=(?<mediaFolder>[\w]+)[\s&]+.+",
"message",".+cmd=(?<command>[\w]+).+"]
}

we have three different combinations of logs in the same logfile.

Now we are trying to use elasticsearch pipeline facility with grok processor. But in the pipeline grok processor we can list multiple patterns but only first match returns. What should be the best way to achieve the same functionality. Help is greatly appreciated.

Thanks
Foysal

[talevy (Tal Levy)]

You're absolutely right that this feature does not exist. Ingest, in effect, has break_on_match => true by default.

If all can potentially match, maybe split them up in separate grok processors? so that they are all applied

let me know if that would result in the behavior you're looking for

[mytzusky]

Was just about to ask the same thing, any plans on including this option in the Grok processor ?

Also, for wiki, a possible solution I've used is to wrap each pattern in a ( )* so it ignores missing matches, in his example:

{
        "grok": {
            "field": "message",
            "patterns": ["%{VENDOR}%{FOLDER}%{COMMAND}"],
            "pattern_definitions" : {
                "VENDOR" : "(.+mv=(?<mediaVendor>[\w]+)[\s&]+.+)*",
                "FOLDER" : "(.+mf=(?<mediaFolder>[\w]+)[\s&]+.+)*",
                "COMMAND" : "(.+cmd=(?<command>[\w]+).+)*"
            },
            "ignore_failure": true
        }
    },

I'm not sure what's the performance compared to having different grok processors but it works.