# Kubernetes Filebeat autodiscover does not find containers

**URL:** <https://discuss.elastic.co/t/kubernetes-filebeat-autodiscover-does-not-find-containers/316713>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [October 16, 2022, 9:17pm UTC](https://discuss.elastic.co/t/kubernetes-filebeat-autodiscover-does-not-find-containers/316713 "2022-10-16T21:17:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![akiks](https://avatars.discourse-cdn.com/v4/letter/a/e9c0ed/32.png) [@akiks](https://discuss.elastic.co/u/akiks)\
**Post date:** [October 16, 2022, 9:17pm UTC](https://discuss.elastic.co/t/kubernetes-filebeat-autodiscover-does-not-find-containers/316713/1 "2022-10-16T21:17:53Z")

</div>

I'm trying to let filebeat collect logs of containers running within Kubernetes using a conditional autodiscover template, but filebeat does never print any of the containers logs.

* * *

Here is an example container whose logs I want to collect:

```yaml
apiVersion: v1
kind: Pod

metadata:
  name: hello
  namespace: elk

spec:
  containers:
  - name: hello
    image: alpine:3.16
    command: ["/bin/sh", "-c", "while true; do echo HELLO WORLD; sleep 1; done"]

```

* * *

I've deployed filebeat within Kubernetes using the Operator and configured a service account.

```yaml
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
  namespace: elk

spec:
  type: filebeat
  version: 8.4.3

  config:
    output.console:
      enabled: true

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition:
                equals:
                   kubernetes.pod.name: "hello"
              config:
                type: container
                paths:
                  - /var/log/containers/*-${data.kubernetes.container.id}.log

  daemonSet:
    podTemplate:
      metadata:
        labels:
          co.elastic.logs/enabled: "false"
        annotations:
          harvestme: "false"
      spec:
        serviceAccount: elastic-beat-filebeat
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        hostNetwork: true
        dnsPolicy: ClusterFirstWithHostNet
        containers:
          - name: filebeat
            env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlogpods
                mountPath: /var/log/pods
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers

        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-beat-filebeat

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-beat-autodiscover-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elastic-beat-autodiscover
subjects:
  - kind: ServiceAccount
    name: elastic-beat-filebeat
    namespace: elk

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-beat-autodiscover
rules:
- apiGroups: [""]
  resources:
    - nodes
    - namespaces
    - events
    - pods
  verbs:
    - get
    - list
    - watch

```

* * *

To make sure that filebeat is able to use the Kubernetes API and to see how the event looks against which I'll be defining conditions, I've first started it with `-d autodiscover`. It did log messages related to the pod:

```auto
{
  "log.level": "debug",
  "@timestamp": "2022-10-16T20:45:33.245Z",
  "log.logger": "autodiscover",
  "log.origin": {
    "file.name": "autodiscover/autodiscover.go",
    "file.line": 182
  },
  "message": "Got a start event.",
  "service.name": "filebeat",
  "autodiscover.event": {
    "config": [],
    "host": "10.244.2.5",
    "id": "ec9af396-f8e1-459a-8c37-ad291d3aca6b.hello",
    "kubernetes": {
      "container": {
        "id": "898e11855ce2ea22d674b4636738b36bbf8a6075b57a9942412beef42f2cabe8",
        "image": "alpine:3.16",
        "name": "hello",
        "runtime": "containerd"
      },
      "labels": {
        "argocd_argoproj_io/instance": "elk"
      },
      "namespace": "elk",
      "namespace_uid": "bef177f4-ea8c-407b-80b8-13742240c20f",
      "node": {
        ...
      },
      "pod": {
        "ip": "10.244.2.5",
        "name": "hello",
        "uid": "ec9af396-f8e1-459a-8c37-ad291d3aca6b"
      }
    },
    "meta": {
      "container": {
        "id": "898e11855ce2ea22d674b4636738b36bbf8a6075b57a9942412beef42f2cabe8",
        "image": {
          "name": "alpine:3.16"
        },
        "runtime": "containerd"
      },
      "kubernetes": {
        "container": {
          "name": "hello"
        },
        "labels": {
          "argocd_argoproj_io/instance": "elk"
        },
        "namespace": "elk",
        "namespace_labels": {
          "kubernetes_io/metadata_name": "elk"
        },
        "namespace_uid": "bef177f4-ea8c-407b-80b8-13742240c20f",
        "node": {
          ...
        },
        "pod": {
          "ip": "10.244.2.5",
          "name": "hello",
          "uid": "ec9af396-f8e1-459a-8c37-ad291d3aca6b"
        }
      }
    },
    "provider": "986b382b-8538-456b-831e-54b300d9ddf4",
    "start": true
  },
  "ecs.version": "1.6.0"
}

```

* * *

I've also ensures that my test-containers log-file is properly mounted into the filebeat container:

```auto
user@filebeat-container $ tail -f /var/log/containers/*-898e11855ce2ea22d674b4636738b36bbf8a6075b57a9942412beef42f2cabe8.log
2022-10-16T23:13:49.545317735+02:00 stdout F HELLO WORLD
2022-10-16T23:13:50.547030092+02:00 stdout F HELLO WORLD
2022-10-16T23:13:51.549656072+02:00 stdout F HELLO WORLD
2022-10-16T23:13:52.56234507+02:00 stdout F HELLO WORLD
2022-10-16T23:13:53.55275043+02:00 stdout F HELLO WORLD
2022-10-16T23:13:54.554603841+02:00 stdout F HELLO WORLD
2022-10-16T23:13:55.555910381+02:00 stdout F HELLO WORLD
...

```

* * *

I've now started filebeat normally (without debug logging). It does not print my containers logs. It only logs the "Non-zero metrics in the last 30s" every few seconds.

```auto
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.669Z","log.origin":{"file.name":"instance/beat.go","file.line":707},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.670Z","log.origin":{"file.name":"instance/beat.go","file.line":715},"message":"Beat ID: b07883a5-a478-4d6c-85d8-26b4ccf7c5e2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.671Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.671Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1081},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"b07883a5-a478-4d6c-85d8-26b4ccf7c5e2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.672Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1090},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"c2f2aba479653563dbaabefe0f86f5579708ec94","libbeat":"8.4.3","time":"2022-09-27T15:24:56.000Z","version":"8.4.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.672Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1093},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.12"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.677Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1097},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-09-11T09:17:14Z","containerized":true,"name":"<...>","ip":["127.0.0.1/8","<...>/24","172.17.0.1/16","172.18.0.1/16","172.19.0.1/16","10.244.2.0/32","10.244.2.1/24"],"kernel_version":"4.19.0-20-amd64","mac":[...],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.678Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1126},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-10-16T20:54:55.700Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.678Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: filebeat; Version: 8.4.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.688Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: <...>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.689Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-10-16T20:54:56.689Z","log.origin":{"file.name":"beater/filebeat.go","file.line":163},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.690Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.690Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":143},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.684Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=18308","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-10-16T20:54:57.684Z","log.origin":{"file.name":"beater/filebeat.go","file.line":289},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.685Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 42","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.685Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.685Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.706Z","log.logger":"autodiscover.pod","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node <...> discovered by NODE_NAME environment variable","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.713Z","log.logger":"autodiscover","log.origin":{"file.name":"autodiscover/autodiscover.go","file.line":118},"message":"Starting autodiscover manager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:55:26.699Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"/"},"cpuacct":{"id":"/","total":{"ns":1803463387}},"memory":{"id":"/","mem":{"limit":{"bytes":209715200},"usage":{"bytes":94486528}}}},"cpu":{"system":{"ticks":270,"time":{"ms":270}},"total":{"ticks":1750,"time":{"ms":1750},"value":1750},"user":{"ticks":1480,"time":{"ms":1480}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b7538c1-2b39-445e-b1f5-42bcaf1d6fe7","name":"filebeat","uptime":{"ms":30205},"version":"8.4.3"},"memstats":{"gc_next":19003472,"memory_alloc":10052600,"memory_sys":39076872,"memory_total":98097584,"rss":143085568},"runtime":{"goroutines":58}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"console"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":4.37,"15":4.25,"5":4.21,"norm":{"1":0.5463,"15":0.5313,"5":0.5263}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:55:56.698Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":28669487}},"memory":{"mem":{"usage":{"bytes":167936}}}},"cpu":{"system":{"ticks":290,"time":{"ms":20}},"total":{"ticks":1780,"time":{"ms":30},"value":1780},"user":{"ticks":1490,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b7538c1-2b39-445e-b1f5-42bcaf1d6fe7","uptime":{"ms":60206},"version":"8.4.3"},"memstats":{"gc_next":19003472,"memory_alloc":10159560,"memory_total":98204544,"rss":143085568},"runtime":{"goroutines":58}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":4.91,"15":4.31,"5":4.4,"norm":{"1":0.6138,"15":0.5387,"5":0.55}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:56:26.696Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":19141517}},"memory":{"mem":{"usage":{"bytes":20480}}}},"cpu":{"system":{"ticks":300,"time":{"ms":10}},"total":{"ticks":1800,"time":{"ms":20},"value":1800},"user":{"ticks":1500,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b7538c1-2b39-445e-b1f5-42bcaf1d6fe7","uptime":{"ms":90206},"version":"8.4.3"},"memstats":{"gc_next":19003472,"memory_alloc":10487104,"memory_total":98532088,"rss":143085568},"runtime":{"goroutines":58}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":3.95,"15":4.26,"5":4.23,"norm":{"1":0.4938,"15":0.5325,"5":0.5288}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:56:56.702Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":21410114}},"memory":{"mem":{"usage":{"bytes":24576}}}},"cpu":{"system":{"ticks":300},"total":{"ticks":1810,"time":{"ms":10},"value":1810},"user":{"ticks":1510,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b7538c1-2b39-445e-b1f5-42bcaf1d6fe7","uptime":{"ms":120205},"version":"8.4.3"},"memstats":{"gc_next":19003472,"memory_alloc":10723688,"memory_total":98768672,"rss":143085568},"runtime":{"goroutines":58}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":6.18,"15":4.41,"5":4.69,"norm":{"1":0.7725,"15":0.5513,"5":0.5863}}}},"ecs.version":"1.6.0"}}
...

```

Does anybody have any clue what could be wrong with my setup?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 13, 2022, 11:18pm UTC](https://discuss.elastic.co/t/kubernetes-filebeat-autodiscover-does-not-find-containers/316713/2 "2022-11-13T23:18:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.