I'm trying to let filebeat collect logs of containers running within Kubernetes using a conditional autodiscover template, but filebeat does never print any of the containers logs.
Here is an example container whose logs I want to collect:
apiVersion: v1
kind: Pod
metadata:
name: hello
namespace: elk
spec:
containers:
- name: hello
image: alpine:3.16
command: ["/bin/sh", "-c", "while true; do echo HELLO WORLD; sleep 1; done"]
I've deployed filebeat within Kubernetes using the Operator and configured a service account.
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: filebeat
namespace: elk
spec:
type: filebeat
version: 8.4.3
config:
output.console:
enabled: true
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.pod.name: "hello"
config:
type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
daemonSet:
podTemplate:
metadata:
labels:
co.elastic.logs/enabled: "false"
annotations:
harvestme: "false"
spec:
serviceAccount: elastic-beat-filebeat
automountServiceAccountToken: true
securityContext:
runAsUser: 0
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: filebeat
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: elastic-beat-filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: elastic-beat-autodiscover-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: elastic-beat-autodiscover
subjects:
- kind: ServiceAccount
name: elastic-beat-filebeat
namespace: elk
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: elastic-beat-autodiscover
rules:
- apiGroups: [""]
resources:
- nodes
- namespaces
- events
- pods
verbs:
- get
- list
- watch
To make sure that filebeat is able to use the Kubernetes API and to see how the event looks against which I'll be defining conditions, I've first started it with -d autodiscover. It did log messages related to the pod:
{
"log.level": "debug",
"@timestamp": "2022-10-16T20:45:33.245Z",
"log.logger": "autodiscover",
"log.origin": {
"file.name": "autodiscover/autodiscover.go",
"file.line": 182
},
"message": "Got a start event.",
"service.name": "filebeat",
"autodiscover.event": {
"config": [],
"host": "10.244.2.5",
"id": "ec9af396-f8e1-459a-8c37-ad291d3aca6b.hello",
"kubernetes": {
"container": {
"id": "898e11855ce2ea22d674b4636738b36bbf8a6075b57a9942412beef42f2cabe8",
"image": "alpine:3.16",
"name": "hello",
"runtime": "containerd"
},
"labels": {
"argocd_argoproj_io/instance": "elk"
},
"namespace": "elk",
"namespace_uid": "bef177f4-ea8c-407b-80b8-13742240c20f",
"node": {
...
},
"pod": {
"ip": "10.244.2.5",
"name": "hello",
"uid": "ec9af396-f8e1-459a-8c37-ad291d3aca6b"
}
},
"meta": {
"container": {
"id": "898e11855ce2ea22d674b4636738b36bbf8a6075b57a9942412beef42f2cabe8",
"image": {
"name": "alpine:3.16"
},
"runtime": "containerd"
},
"kubernetes": {
"container": {
"name": "hello"
},
"labels": {
"argocd_argoproj_io/instance": "elk"
},
"namespace": "elk",
"namespace_labels": {
"kubernetes_io/metadata_name": "elk"
},
"namespace_uid": "bef177f4-ea8c-407b-80b8-13742240c20f",
"node": {
...
},
"pod": {
"ip": "10.244.2.5",
"name": "hello",
"uid": "ec9af396-f8e1-459a-8c37-ad291d3aca6b"
}
}
},
"provider": "986b382b-8538-456b-831e-54b300d9ddf4",
"start": true
},
"ecs.version": "1.6.0"
}
I've also ensures that my test-containers log-file is properly mounted into the filebeat container:
user@filebeat-container $ tail -f /var/log/containers/*-898e11855ce2ea22d674b4636738b36bbf8a6075b57a9942412beef42f2cabe8.log
2022-10-16T23:13:49.545317735+02:00 stdout F HELLO WORLD
2022-10-16T23:13:50.547030092+02:00 stdout F HELLO WORLD
2022-10-16T23:13:51.549656072+02:00 stdout F HELLO WORLD
2022-10-16T23:13:52.56234507+02:00 stdout F HELLO WORLD
2022-10-16T23:13:53.55275043+02:00 stdout F HELLO WORLD
2022-10-16T23:13:54.554603841+02:00 stdout F HELLO WORLD
2022-10-16T23:13:55.555910381+02:00 stdout F HELLO WORLD
...
I've now started filebeat normally (without debug logging). It does not print my containers logs. It only logs the "Non-zero metrics in the last 30s" every few seconds.
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.669Z","log.origin":{"file.name":"instance/beat.go","file.line":707},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.670Z","log.origin":{"file.name":"instance/beat.go","file.line":715},"message":"Beat ID: b07883a5-a478-4d6c-85d8-26b4ccf7c5e2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.671Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.671Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1081},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"b07883a5-a478-4d6c-85d8-26b4ccf7c5e2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.672Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1090},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"c2f2aba479653563dbaabefe0f86f5579708ec94","libbeat":"8.4.3","time":"2022-09-27T15:24:56.000Z","version":"8.4.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.672Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1093},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.12"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.677Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1097},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-09-11T09:17:14Z","containerized":true,"name":"<...>","ip":["127.0.0.1/8","<...>/24","172.17.0.1/16","172.18.0.1/16","172.19.0.1/16","10.244.2.0/32","10.244.2.1/24"],"kernel_version":"4.19.0-20-amd64","mac":[...],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.678Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1126},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-10-16T20:54:55.700Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.678Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: filebeat; Version: 8.4.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.688Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: <...>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.689Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-10-16T20:54:56.689Z","log.origin":{"file.name":"beater/filebeat.go","file.line":163},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.690Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:56.690Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":143},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.684Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=18308","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-10-16T20:54:57.684Z","log.origin":{"file.name":"beater/filebeat.go","file.line":289},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.685Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 42","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.685Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.685Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.706Z","log.logger":"autodiscover.pod","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node <...> discovered by NODE_NAME environment variable","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:54:57.713Z","log.logger":"autodiscover","log.origin":{"file.name":"autodiscover/autodiscover.go","file.line":118},"message":"Starting autodiscover manager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-16T20:55:26.699Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"/"},"cpuacct":{"id":"/","total":{"ns":1803463387}},"memory":{"id":"/","mem":{"limit":{"bytes":209715200},"usage":{"bytes":94486528}}}},"cpu":{"system":{"ticks":270,"time":{"ms":270}},"total":{"ticks":1750,"time":{"ms":1750},"value":1750},"user":{"ticks":1480,"time":{"ms":1480}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b7538c1-2b39-445e-b1f5-42bcaf1d6fe7","name":"filebeat","uptime":{"ms":30205},"version":"8.4.3"},"memstats":{"gc_next":19003472,"memory_alloc":10052600,"memory_sys":39076872,"memory_total":98097584,"rss":143085568},"runtime":{"goroutines":58}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"console"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":4.37,"15":4.25,"5":4.21,"norm":{"1":0.5463,"15":0.5313,"5":0.5263}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:55:56.698Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":28669487}},"memory":{"mem":{"usage":{"bytes":167936}}}},"cpu":{"system":{"ticks":290,"time":{"ms":20}},"total":{"ticks":1780,"time":{"ms":30},"value":1780},"user":{"ticks":1490,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b7538c1-2b39-445e-b1f5-42bcaf1d6fe7","uptime":{"ms":60206},"version":"8.4.3"},"memstats":{"gc_next":19003472,"memory_alloc":10159560,"memory_total":98204544,"rss":143085568},"runtime":{"goroutines":58}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":4.91,"15":4.31,"5":4.4,"norm":{"1":0.6138,"15":0.5387,"5":0.55}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:56:26.696Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":19141517}},"memory":{"mem":{"usage":{"bytes":20480}}}},"cpu":{"system":{"ticks":300,"time":{"ms":10}},"total":{"ticks":1800,"time":{"ms":20},"value":1800},"user":{"ticks":1500,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b7538c1-2b39-445e-b1f5-42bcaf1d6fe7","uptime":{"ms":90206},"version":"8.4.3"},"memstats":{"gc_next":19003472,"memory_alloc":10487104,"memory_total":98532088,"rss":143085568},"runtime":{"goroutines":58}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":3.95,"15":4.26,"5":4.23,"norm":{"1":0.4938,"15":0.5325,"5":0.5288}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-10-16T20:56:56.702Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":21410114}},"memory":{"mem":{"usage":{"bytes":24576}}}},"cpu":{"system":{"ticks":300},"total":{"ticks":1810,"time":{"ms":10},"value":1810},"user":{"ticks":1510,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b7538c1-2b39-445e-b1f5-42bcaf1d6fe7","uptime":{"ms":120205},"version":"8.4.3"},"memstats":{"gc_next":19003472,"memory_alloc":10723688,"memory_total":98768672,"rss":143085568},"runtime":{"goroutines":58}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":6.18,"15":4.41,"5":4.69,"norm":{"1":0.7725,"15":0.5513,"5":0.5863}}}},"ecs.version":"1.6.0"}}
...
Does anybody have any clue what could be wrong with my setup?