Hi, I have trouble when deploying Filebeat in my Kubernetes cluster (v1.21.1) using ECK.
The deployment spec is the following:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: filebeat
spec:
type: filebeat
version: 7.11.2
config:
filebeat:
autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints:
enabled: true
default_config.enabled: false
output.redis:
enabled: true
hosts: ["redis:6379"]
password: topsecret
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
spec:
serviceAccountName: filebeat
automountServiceAccountToken: true
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true # Allows to provide richer host metadata
containers:
- name: filebeat
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
volumeMounts:
- name: var-log-containers
mountPath: /var/log/containers
- name: var-log-pods
mountPath: /var/log/pods
- name: var-lib-docker-containers
mountPath: /var/lib/docker/containers
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumes:
- name: var-log-containers
hostPath:
path: /var/log/containers
- name: var-log-pods
hostPath:
path: /var/log/pods
- name: var-lib-docker-containers
hostPath:
path: /var/lib/docker/containers
As you can see, default_config.enabled: false aims at disabling logs scraping by default on all pods, except for those having a co.elastic.logs/enabled: "true" annotation.
It works well in Kubernetes clusters when the container runtime is docker, however it does not seems to work on clusters with containerd-backed Kubernetes nodes.
After some research, I found out that Filbeat try retrieving logs from the /var/lib/docker/containers directory by default, instead of the standard CRI path /var/log/containers, as shown in the Filebeat log sample below:
Configured paths: [/var/lib/docker/containers/6f0a966ac5da58d62911d8241d93a202a7096aee8c9f2c642b94b3a7ae689313/*-json.log]
So my question is, is it possible to override the logs path used by Filebeat while keeping hints.default_config.enabled to false ?
I tried fixes with templates or appenders, without success as they seem to override the hints block (thus enabling logs collection from all pods), and the doc is quite vague about the underlying inputs generated by the autodiscover provider for Kubernets.
Thanks,