When trying to deploy Filebeat as a Daemonset in k8s we are able to see logs from all the pods running on that particular host. But we would like to add the k8s metadata to filter based on the deployment name as each one has a different Logstash configuration. When adding the k8s annotate section, we are seeing an error where it is trying to look for a pod with the hostname of the worker node. Can you help us identify the issue?
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
stream: "stderr"
include_lines: ['com.example']
processors:
- add_kubernetes_metadata:
Error received:
|2020-03-12T09:11:18.105Z|INFO|kubernetes/util.go:94|kubernetes: Using pod name WORKER153 and namespace kube-system to discover kubernetes node
|2020-03-12T09:11:18.109Z|ERROR|kubernetes/util.go:97|kubernetes: Querying for pod failed with error: pods "WORKER153" not found