How to use grok patterns coming from event data

[whites11]

I have an ELK stack deployed on kubernetes used to collect containers' data. Among all the rest, it is using a grok filter to parse the actual log line based on a pattern.

My wish is to be able to setup this pattern by using an annotation in the kubernetes pod.

I added an annotation called elk-grok-pattern in the pod, configured filebeat in order to forward the annotation and I can get the annotation value as a field in my event in logstash, so far so good.

The problem is that I am unable to use the value of my field as a grok pattern.

The annotation in my pod looks like this:

Annotations:    elk-grok-pattern=%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:status} %{NUMBER:response_time}

The filter I am trying to use is similar to the following:

filter {
  # create a new field called "elk-grok-pattern" from the pod annotation
  mutate {
        rename => { "[kubernetes][annotations][elk-grok-pattern]" => "elk-grok-pattern" }
  }

  grok {
    pattern_definitions => {
      "CUSTOM" => "%{elk-grok-pattern}"
    }
    match => { "log" => "%{CUSTOM}" }
  }
}

Unluckily this leads to an error:

Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{elk-grok-pattern} not defined>

In practice, grok is interpreting my pattern literally, and not evaluating the string content coming from the event.

I also tried using the pattern directly, withoud defining a pattern_definition, like this:

grok {
  match => { "log" => "%{elk-grok-pattern}" }
}

But I get the same exact error.

Is there a way to accomplish my goal?
Any advice or possible workaround would be very appreciated.