# Filebeat autodiscover for Kubernetes uses inconsistent log files path by default

**URL:** <https://discuss.elastic.co/t/filebeat-autodiscover-for-kubernetes-uses-inconsistent-log-files-path-by-default/279834>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [July 28, 2021, 9:54am UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-kubernetes-uses-inconsistent-log-files-path-by-default/279834 "2021-07-28T09:54:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![NicolasCa](https://avatars.discourse-cdn.com/v4/letter/n/e480ec/32.png) [@NicolasCa](https://discuss.elastic.co/u/NicolasCa)\
**Post date:** [July 28, 2021, 9:54am UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-kubernetes-uses-inconsistent-log-files-path-by-default/279834/1 "2021-07-28T09:54:32Z")

</div>

Hi, I have trouble when deploying Filebeat in my Kubernetes cluster (v1.21.1) using ECK.

The deployment spec is the following:

```auto
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 7.11.2
  config:
    filebeat:
      autodiscover:
        providers:
          - type: kubernetes
            node: ${NODE_NAME}
            hints:
              enabled: true
              default_config.enabled: false
    output.redis:
      enabled: true
      hosts: ["redis:6379"]
      password: topsecret
    processors:
      - add_cloud_metadata: {}
      - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        containers:
          - name: filebeat
            securityContext:
              runAsUser: 0
              # If using Red Hat OpenShift uncomment this:
              #privileged: true
            volumeMounts:
              - name: var-log-containers
                mountPath: /var/log/containers
              - name: var-log-pods
                mountPath: /var/log/pods
              - name: var-lib-docker-containers
                mountPath: /var/lib/docker/containers
            env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
        volumes:
          - name: var-log-containers
            hostPath:
              path: /var/log/containers
          - name: var-log-pods
            hostPath:
              path: /var/log/pods
          - name: var-lib-docker-containers
            hostPath:
              path: /var/lib/docker/containers

```

As you can see, `default_config.enabled: false` aims at disabling logs scraping by default on all pods, except for those having a `co.elastic.logs/enabled: "true"` annotation.

It works well in Kubernetes clusters when the container runtime is `docker`, however it does not seems to work on clusters with `containerd`-backed Kubernetes nodes.

After some research, I found out that Filbeat try retrieving logs from the `/var/lib/docker/containers` directory by default, instead of the standard CRI path `/var/log/containers`, as shown in the Filebeat log sample below:

```auto
Configured paths: [/var/lib/docker/containers/6f0a966ac5da58d62911d8241d93a202a7096aee8c9f2c642b94b3a7ae689313/*-json.log]

```

So my question is, is it possible to override the logs path used by Filebeat while keeping `hints.default_config.enabled` to false ?

I tried fixes with `templates` or `appenders`, without success as they seem to override the `hints` block (thus enabling logs collection from all pods), and the doc is quite vague about the underlying inputs generated by the autodiscover provider for Kubernets.

Thanks,

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [July 29, 2021, 8:10am UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-kubernetes-uses-inconsistent-log-files-path-by-default/279834/2 "2021-07-29T08:10:32Z")

</div>

Hi!

This is most probably because of [beats/config.go at 9a4326be0ee18a074abc993b41a5130b643769c7 · elastic/beats · GitHub](https://github.com/elastic/beats/blob/9a4326be0ee18a074abc993b41a5130b643769c7/filebeat/autodiscover/builder/hints/config.go#L33) and something that should be investigated and fixed. Could you please file a GH issue for this so as the team to look into it?

C.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 26, 2021, 10:10am UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-kubernetes-uses-inconsistent-log-files-path-by-default/279834/3 "2021-08-26T10:10:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.