ES grok processor break_on_match => false needed

Elastic Stack Elasticsearch
1

We have a pattern in logstash grok filter like this
grok {
break_on_match => false
match => [ "message",".+mv=(?<mediaVendor>[\w]+)[\s&]+.+",
"message",".+mf=(?<mediaFolder>[\w]+)[\s&]+.+",
"message",".+cmd=(?<command>[\w]+).+"]
}

we have three different combinations of logs in the same logfile.

Now we are trying to use elasticsearch pipeline facility with grok processor. But in the pipeline grok processor we can list multiple patterns but only first match returns. What should be the best way to achieve the same functionality. Help is greatly appreciated.

Thanks
Foysal

2

You're absolutely right that this feature does not exist. Ingest, in effect, has break_on_match => true by default.

If all can potentially match, maybe split them up in separate grok processors? so that they are all applied

let me know if that would result in the behavior you're looking for

3

Was just about to ask the same thing, any plans on including this option in the Grok processor ?

Also, for wiki, a possible solution I've used is to wrap each pattern in a ( )* so it ignores missing matches, in his example:

{
        "grok": {
            "field": "message",
            "patterns": ["%{VENDOR}%{FOLDER}%{COMMAND}"],
            "pattern_definitions" : {
                "VENDOR" : "(.+mv=(?<mediaVendor>[\w]+)[\s&]+.+)*",
                "FOLDER" : "(.+mf=(?<mediaFolder>[\w]+)[\s&]+.+)*",
                "COMMAND" : "(.+cmd=(?<command>[\w]+).+)*"
            },
            "ignore_failure": true
        }
    },

I'm not sure what's the performance compared to having different grok processors but it works.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.