I'm deploy the elastic.co logstash helm chart 7.7.1 to kubernetes.
I'm using a ConfigMap to create an example
file in /usr/share/logstash/pipeline/patterns/
But it seems like it's not being read.
I keep getting an error about the pattern not existing, see error, ConfigMap, and answers.yaml below.
Error Message:
Pipeline aborted due to error {
:pipeline_id=>"main",
:exception=>#<Grok::PatternError: pattern %{TS:[@metadata][ts]} not defined>,
:backtrace=>[
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'",
"org/jruby/RubyKernel.java:1442:in `loop'",
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'",
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:288:in `block in register'",
"org/jruby/RubyArray.java:1809:in `each'",
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:282:in `block in register'",
"org/jruby/RubyHash.java:1415:in `each'",
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:277:in `register'",
"org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:216:in `block in register_plugins'",
"org/jruby/RubyArray.java:1809:in `each'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:215:in `register_plugins'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:521:in `maybe_setup_out_plugins'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `start_workers'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170:in `run'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125:in `block in start'"],
"pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"],
:thread=>"#<Thread:0x27793dfe run>"
}
Below is the configmap I'm using to create the patterns directory and files:
apiVersion: v1
kind: ConfigMap
metadata:
name: patterns-config
data:
example: |-
TS \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}
NDQ [^\"]+
BKT [^\]]+
BKTNS [^\] ]+
NCOLON [^:]+
Below is a portion of the answers file I'm using:
persistence:
enabled: false
service:
annotations: {}
type: ClusterIP
ports:
- name: beats
port: 5044
protocol: TCP
targetPort: 5044
- name: http
port: 8080
protocol: TCP
targetPort: 8080
secretMounts:
- name: logstash-certificates-pem
secretName: logstash-certificates-pem
path: /usr/share/logstash/config/certs
extraVolumeMounts: |
- name: patterns
mountPath: "/usr/share/logstash/pipeline/patterns"
readOnly: true
extraVolumes: |
- name: patterns
configMap:
name: patterns-config
logstashPipeline:
logstash.conf: |
input {
beats {
port => "5044"
}
}
filter {
if [fields][log_type] == "example" {
grok {
patterns_dir => ["./patterns"]
match => {
"message" => "^%{TS:[@metadata][ts]} \[%{BKT:example_level}\] \[%{BKTNS:example_tid}\s*\] \[%{BKTNS:example_library}\s*\] - %{GREEDYDATA:msg}"
}
}
}
output {
stdout { codec => rubydebug }
}