Passing custom regex inside grok filter

[Flavio1]

Hi everybody,
During these days i'm trying to implement a custom regex system configuration in order to have a single point, outside pipelines, in which i can make crud operations on regexes. I tried this solution:

mutate {
            add_field => {
                          "regex_status" => "specific_status"                        
                          "default_status" => "default_status"
            }
        }       
        translate {
            field => "regex_status"
            destination => "[regex_data]"
            dictionary_path => "C:/lookup-regex.json"
            add_field => { "status_exists" => "true" }
        }
        if ([status_exists] == "true") {
          grok {
             match => [ "message", "%{[regex_data][status]}" ]
          }
        }
        else{
          translate {
                field => "default_timestamp"
                destination => "[regex_data]"
                dictionary_path => "c:/lookup-regex.json"   
          }

and this is the lookup-regex.json:

{
    "default_status":".*HTTPv2.0\\/\\d\\.\\d\\\"\\s(?<status>\\d{3})\\s",
    "specific_status": {
        "status":".*HTTP\\/\\d\\.\\d\\\"\\s(?<status>\\d{3})\\s"
    }
}

It seems that during the parsing of the pipeline the logstash debugger gives me that pipeline stop worked because %{[regex_data][status]} it's not a regex. Is this any way to accomplish this behaviour? Essentialy, i would like to know if i can pass string of regex like variable in grok filters. Thx in advance.

[leandrojmp (Leandro Pereira)]

I'm not sure if this approach would work and it seems overcomplicated when you compare with the approach in the documentation to use custom patterns.

You could try putting your custom regex expressions in a file and use the patterns_dir option in your grok filter.

For example, you could create a file named CUSTOM-REGEX with the following content.

DEFAULTSTATUS .*HTTPv2.0\\/\\d\\.\\d\\\"\\s(?<status>\\d{3})\\s
SPECIFICSTATUS .*HTTP\\/\\d\\.\\d\\\"\\s(?<status>\\d{3})\\s

Then you would use this file in your grok config.

grok {
    patterns_dir => ["/path/to/your/custom/regex/file"]
    match => { "message", "%{DEFAULTSTATUS:field_name}" }
}