Filebeat version: 7.5.2
I am trying to get my application running on Kubernetes with the ELK stack to do logging. The application is written in Java so I need to be able to able to ingest multiline stack traces as a single log message, I have a regex in Filebeat that does this. I want to apply this multiline filtering only to pods with the Kubernetes app label "my-app", the logs for everything else should be ingested regularly.
I am having a lot of trouble setting up autodiscovery, my config (from my k8s filebeat manifest) is below:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.labels.app: "my-app"
config:
- type: container
#containers.ids:
# - "${data.docker.container.id}"
# - "${data.kubernetes.container.id}" #"*"
paths:
- /var/log/container/*-${data.kubernetes.container.id}.log
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
filebeat.config:
#inputs: # prospectors formerly
# Mounted `filebeat-prospectors` configmap, keep it in the prospector dir for now TODO
#path: ${path.config}/prospectors.d/*.yml
# Reload prospectors configs as they change:
#reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
# all modules are disabled currently
reload.enabled: false
output.logstash:
hosts: ['logstash-service:5044']
multiline.match: after
I tried adding a paths field and I tried using containers.ids. I still can't see my logs in Kibana using the above config. When using "containers.ids" Filebeat was complaining that a "paths" needs to be set, I set it and it still doesn't work. My cluster is working correctly since using the config below works but ignores other non-"my-app" logs that I want:
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
processors:
- add_kubernetes_metadata:
in_cluster: true
If autodiscovery is not the way to do this then what is? I want a multiline filter applied to my Spring app logs (so stack traces get grouped as one message) and I want default (no filtering) applied to all other logs. Potentially I may want to add more custom rules in the future too. I have like 15 or so services running already ideally I don't want to add a rule for all of them, only the ones for which I don't want the default behavior for logging.