Filebeat autodiscover for Kubernetes uses inconsistent log files path by default

Elastic Stack Beats
,
1

Hi, I have trouble when deploying Filebeat in my Kubernetes cluster (v1.21.1) using ECK.

The deployment spec is the following:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 7.11.2
  config:
    filebeat:
      autodiscover:
        providers:
          - type: kubernetes
            node: ${NODE_NAME}
            hints:
              enabled: true
              default_config.enabled: false
    output.redis:
      enabled: true
      hosts: ["redis:6379"]
      password: topsecret
    processors:
      - add_cloud_metadata: {}
      - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        containers:
          - name: filebeat
            securityContext:
              runAsUser: 0
              # If using Red Hat OpenShift uncomment this:
              #privileged: true
            volumeMounts:
              - name: var-log-containers
                mountPath: /var/log/containers
              - name: var-log-pods
                mountPath: /var/log/pods
              - name: var-lib-docker-containers
                mountPath: /var/lib/docker/containers
            env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
        volumes:
          - name: var-log-containers
            hostPath:
              path: /var/log/containers
          - name: var-log-pods
            hostPath:
              path: /var/log/pods
          - name: var-lib-docker-containers
            hostPath:
              path: /var/lib/docker/containers

As you can see, default_config.enabled: false aims at disabling logs scraping by default on all pods, except for those having a co.elastic.logs/enabled: "true" annotation.

It works well in Kubernetes clusters when the container runtime is docker, however it does not seems to work on clusters with containerd-backed Kubernetes nodes.

After some research, I found out that Filbeat try retrieving logs from the /var/lib/docker/containers directory by default, instead of the standard CRI path /var/log/containers, as shown in the Filebeat log sample below:

Configured paths: [/var/lib/docker/containers/6f0a966ac5da58d62911d8241d93a202a7096aee8c9f2c642b94b3a7ae689313/*-json.log]

So my question is, is it possible to override the logs path used by Filebeat while keeping hints.default_config.enabled to false ?

I tried fixes with templates or appenders, without success as they seem to override the hints block (thus enabling logs collection from all pods), and the doc is quite vague about the underlying inputs generated by the autodiscover provider for Kubernets.

Thanks,

2

Hi!

This is most probably because of beats/config.go at 9a4326be0ee18a074abc993b41a5130b643769c7 · elastic/beats · GitHub and something that should be investigated and fixed. Could you please file a GH issue for this so as the team to look into it?

C.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.