Hello everyone, I'm trying to implement autodiscover with filebeat using ECK. Basic feature at the moment, using :
config:
filebeat.autodiscover:
providers:
- type: kubernetes
host: ${HOSTNAME} #deprecated
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/log/containers/*-${data.container.id}.log
#filebeat.inputs:
#- type: container
# paths:
# - /var/log/containers/*.log
daemonSet:
podTemplate:
spec:
serviceAccount: elastic-beat-filebeat-quickstart
automountServiceAccountToken: true
securityContext:
runAsUser: 0
containers:
- name: filebeat
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
My stack is up and running, no errors logs in my filebeat pods. I have data harvested in elasticsearch using :
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
With the autodiscover config above (took it right from the documentation here Hints based autodiscover | Filebeat Reference [7.12] | Elastic), I don't have any data sent to elasticsearch.
Here is the startup log from my filebeat pod :
2021-04-20T18:26:15.669Z INFO instance/beat.go:660 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2021-04-20T18:26:15.669Z INFO instance/beat.go:668 Beat ID: c25a1666-1d10-401c-8bac-d17b629ccb69
2021-04-20T18:26:15.671Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-04-20T18:26:15.671Z INFO [beat] instance/beat.go:996 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "c25a1666-1d10-401c-8bac-d17b629ccb69"}}}
2021-04-20T18:26:15.671Z INFO [beat] instance/beat.go:1005 Build info {"system_info": {"build": {"commit": "08e20483a651ea5ad60115f68ff0e53e6360573a", "libbeat": "7.12.0", "time": "2021-03-18T06:16:51.000Z", "version": "7.12.0"}}}
2021-04-20T18:26:15.671Z INFO [beat] instance/beat.go:1008 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.15.8"}}}
2021-04-20T18:26:15.673Z INFO [beat] instance/beat.go:1012 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-04-17T23:08:23Z","containerized":true,"name":"quickstart-beat-filebeat-9xpgf","ip":["127.0.0.1/8","::1/128","10.244.13.168/32","fe80::bc05:59ff:fef3:f328/64"],"kernel_version":"4.15.0-142-generic","mac":["be:05:59:f3:f3:28"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"44ccc339d95bd51386fcfc5d8f041927"}}}
2021-04-20T18:26:15.673Z INFO [beat] instance/beat.go:1041 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 6, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-04-20T18:26:13.340Z"}}}
2021-04-20T18:26:15.673Z INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.12.0
2021-04-20T18:26:15.674Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.12.0' as ILM is enabled.
2021-04-20T18:26:15.674Z WARN [cfgwarn] tlscommon/config.go:101 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2021-04-20T18:26:15.674Z INFO eslegclient/connection.go:99 elasticsearch url: https://quickstart-es-http.elastic.svc:9200
2021-04-20T18:26:15.676Z INFO [publisher] pipeline/module.go:113 Beat name: quickstart-beat-filebeat-9xpgf
2021-04-20T18:26:15.768Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2021-04-20T18:26:15.866Z INFO instance/beat.go:468 filebeat start running.
2021-04-20T18:26:15.872Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=22628509
2021-04-20T18:26:18.867Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=22636372
2021-04-20T18:26:18.870Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 108
2021-04-20T18:26:18.871Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 0
2021-04-20T18:26:18.871Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0
2021-04-20T18:26:18.871Z WARN [cfgwarn] kubernetes/config.go:84 DEPRECATED: `host` will be deprecated, use `node` instead Will be removed in version: 8.0
2021-04-20T18:26:18.968Z WARN [cfgwarn] kubernetes/config.go:84 DEPRECATED: `host` will be deprecated, use `node` instead Will be removed in version: 8.0
2021-04-20T18:26:18.968Z INFO [autodiscover.pod] kubernetes/util.go:99 kubernetes: Using node quickstart-beat-filebeat-9xpgf provided in the config
2021-04-20T18:26:18.968Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager
2021-04-20T18:26:45.774Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":10000}},"id":"cri-containerd-acbf18d343695bbd22c6111ca646a710e0dffb2cadde7b217ebe787c9bb59e74.scope","stats":{"periods":97,"throttled":{"ns":11009093910,"periods":62}}},"cpuacct":{"id":"cri-containerd-acbf18d343695bbd22c6111ca646a710e0dffb2cadde7b217ebe787c9bb59e74.scope","total":{"ns":658774395}},"memory":{"id":"cri-containerd-acbf18d343695bbd22c6111ca646a710e0dffb2cadde7b217ebe787c9bb59e74.scope","mem":{"limit":{"bytes":209715200},"usage":{"bytes":31334400}}}},"cpu":{"system":{"ticks":100,"time":{"ms":101}},"total":{"ticks":610,"time":{"ms":619},"value":610},"user":{"ticks":510,"time":{"ms":518}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"f4e17663-b3b5-4e1a-b331-61a90e827eb9","uptime":{"ms":32197}},"memstats":{"gc_next":20554544,"memory_alloc":12610016,"memory_sys":76235776,"memory_total":67802232,"rss":77000704},"runtime":{"goroutines":52}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":2.72,"15":1.45,"5":1.81,"norm":{"1":0.34,"15":0.1813,"5":0.2263}}}}}}
Does this configuration should harvest log from all pods (even without co.elastic.logs/* annotations in them) ? If so, what could be the misconfiguration ?