I'm running ELK 8.15.0 on Kubernetes using ECK operator.
My cluster-wide filebeat configuration set in a beat.k8s.elastic.co/v1beta1 Beat resource is the following:
filebeat:
autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
add_resource_metadata:
node:
enabled: false
cronjob: true
deployment: true
hints:
enabled: true
default_config:
enabled: false
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
I override it for specific pods using annotations:
co.elastic.logs.app/enabled: 'true'
co.elastic.logs.app/multiline.type: 'pattern'
co.elastic.logs.app/multiline.pattern: '^[0-9:.,TZ-]{24} '
co.elastic.logs.app/multiline.negate: 'true'
co.elastic.logs.app/multiline.match: 'after'
co.elastic.logs.app/processors.1.dissect.tokenizer: '%{@timestamp} %{log.level} [%{log.logger}] (%{process.thread.name}) %{message}'
co.elastic.logs.app/processors.1.dissect.target_prefix: ''
co.elastic.logs.app/processors.1.dissect.overwrite_keys: 'true'
co.elastic.logs.app/processors.1.dissect.trim_values: 'right'
The thing that I am trying to do is running decode_json_fields
processor on log entries with a specific log.logger
value.
As I understand, I need to add the following processor at index 2:
- decode_json_fields:
when:
equals:
log.logger: webservices
fields:
- message
target: webservices
process_array: true
max_depth: 10
However, I don't think the when equals
condition can be expressed using an annotation because log.logger
field name contains a dot.
I was trying to work around that using co.elsastic.logs/raw
annotation but I found out that no matter what I put into it, no logs were collected from the pod, ie the global enabled: false
configuration was in effect. Before I get to adding the new processor I need to get log collection to work again.
When trying to narrow down the problem I found that the following works (logs are collected):
co.elastic.logs/enabled: "true"
But none of the following works (logs are not collected):
co.elastic.logs/raw: "{\"enabled\":\"true\"}"
Hints based autodiscover | Filebeat Reference [8.17] | Elastic uses an array in the example:
co.elastic.logs/raw: "[{\"enabled\":\"true\"}]"
Trying to replicate the content of the global configuration:
co.elastic.logs/raw: "{\"enabled\":\"true\",\"type\":\"container\",\"paths\":[\"/var/log/containers/*${data.kubernetes.container.id}.log\"]}"
I have a suspicion that it might be an character escaping problem, so to be clear I'm showing the annotations above as they appear in the manifest (rendered by Helm) before being applied to the cluster.
Am I making some obvious mistake?
How can I debug that? Can I somehow list live input definitions in the filebeat agent pods running in the cluster? Can I bump logging level somewhere to get information how the hints are being processed?
Any tips appreciated, thanks in advance!