How to create patterns_dir in elastic logstash helm chart?

I'm deploy the elastic.co logstash helm chart 7.7.1 to kubernetes.
I'm using a ConfigMap to create an example file in /usr/share/logstash/pipeline/patterns/
But it seems like it's not being read.
I keep getting an error about the pattern not existing, see error, ConfigMap, and answers.yaml below.

Error Message:

Pipeline aborted due to error {
    :pipeline_id=>"main", 
    :exception=>#<Grok::PatternError: pattern %{TS:[@metadata][ts]} not defined>, 
    :backtrace=>[
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'", 
        "org/jruby/RubyKernel.java:1442:in `loop'", 
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'", 
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:288:in `block in register'", 
        "org/jruby/RubyArray.java:1809:in `each'", 
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:282:in `block in register'", 
        "org/jruby/RubyHash.java:1415:in `each'", 
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:277:in `register'", 
        "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", 
        "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:216:in `block in register_plugins'", 
        "org/jruby/RubyArray.java:1809:in `each'", 
        "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:215:in `register_plugins'", 
        "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:521:in `maybe_setup_out_plugins'", 
        "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `start_workers'", 
        "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170:in `run'", 
        "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125:in `block in start'"], 
        "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], 
        :thread=>"#<Thread:0x27793dfe run>"
}

Below is the configmap I'm using to create the patterns directory and files:

apiVersion: v1
kind: ConfigMap
metadata:
  name: patterns-config
data: 
  example: |-
    TS \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}
    NDQ [^\"]+
    BKT  [^\]]+
    BKTNS  [^\] ]+
    NCOLON [^:]+

Below is a portion of the answers file I'm using:

persistence:
  enabled: false

service:
  annotations: {}
  type: ClusterIP
  ports:
    - name: beats
      port: 5044
      protocol: TCP
      targetPort: 5044
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080

secretMounts:
  - name: logstash-certificates-pem
    secretName: logstash-certificates-pem
    path: /usr/share/logstash/config/certs

extraVolumeMounts: |
  - name: patterns
    mountPath: "/usr/share/logstash/pipeline/patterns"
    readOnly: true

extraVolumes: |
  - name: patterns
    configMap:
      name: patterns-config

logstashPipeline:
  logstash.conf: |
    input {
      beats {
        port => "5044"
      }
    }

    filter {
      if [fields][log_type] == "example" {
        grok {
          patterns_dir => ["./patterns"]
          match => {
            "message" => "^%{TS:[@metadata][ts]} \[%{BKT:example_level}\] \[%{BKTNS:example_tid}\s*\] \[%{BKTNS:example_library}\s*\] - %{GREEDYDATA:msg}"
          }
        }
     }
     output {
      stdout { codec => rubydebug }
     }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.