Hi,
I have a Kubernetes test cluster with Filebeat running as Daemonset and a Redis container I want to monitor.
Filebeat log shows it can find the container because the logs path (/var/log/containers/*-20845e0aa57813e0796823e28ddaf92b6bd9d0e84d35e3affec7dd492043d10c.log) contains the container ID.
This is my Filebeats configuration:
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.container.image: "redis"
config:
- module: redis
log:
input:
type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
slowlog:
enabled: true
var.hosts: ["148.187.106.204:32527"]
processors:
- add_cloud_metadata:
- add_kubernetes_metadata:
- add_docker_metadata:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:148.187.106.204}:${ELASTICSEARCH_PORT:32341}']
These are Filebeats logs:
2020-10-05T23:56:03.244Z INFO instance/beat.go:640 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-10-05T23:56:03.346Z INFO instance/beat.go:648 Beat ID: 23ffb203-c72a-43bc-b5b8-0e35a079366b
2020-10-05T23:56:03.349Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-10-05T23:56:03.349Z INFO [beat] instance/beat.go:976 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "23ffb203-c72a-43bc-b5b8-0e35a079366b"}}}
2020-10-05T23:56:03.349Z INFO [beat] instance/beat.go:985 Build info {"system_info": {"build": {"commit": "ad823eca4cc74439d1a44351c596c12ab51054f5", "libbeat": "7.9.1", "time": "2020-09-01T19:58:51.000Z", "version": "7.9.1"}}}
2020-10-05T23:56:03.349Z INFO [beat] instance/beat.go:988 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":72,"version":"go1.14.7"}}}
2020-10-05T23:56:03.353Z INFO [beat] instance/beat.go:992 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-31T13:52:36Z","containerized":true,"name":"filebeat-deployment-2ztkx","ip":["127.0.0.1/8","::1/128","10.233.64.27/18","fe80::f5:88ff:fe30:fa88/64"],"kernel_version":"5.4.0-42-generic","mac":["02:f5:88:30:fa:88"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-10-05T23:56:03.354Z INFO [beat] instance/beat.go:1021 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-10-05T23:56:02.530Z"}}}
2020-10-05T23:56:03.354Z INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.9.1
2020-10-05T23:56:03.354Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.9.1' as ILM is enabled.
2020-10-05T23:56:03.354Z INFO eslegclient/connection.go:99 elasticsearch url: http://148.187.106.204:32341
2020-10-05T23:56:03.356Z INFO [publisher] pipeline/module.go:113 Beat name: filebeat-deployment-2ztkx
2020-10-05T23:56:03.360Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-10-05T23:56:03.360Z INFO instance/beat.go:450 filebeat start running.
2020-10-05T23:56:03.361Z INFO add_kubernetes_metadata/kubernetes.go:71 add_kubernetes_metadata: kubernetes env detected, with version: v1.18.5
2020-10-05T23:56:03.361Z INFO [kubernetes] kubernetes/util.go:114 kubernetes: Using pod name filebeat-deployment-2ztkx and namespace msopena to discover kubernetes node {"libbeat.processor": "add_kubernetes_metadata"}
2020-10-05T23:56:03.405Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2020-10-05T23:56:03.405Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2020-10-05T23:56:03.405Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2020-10-05T23:56:03.405Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 0
2020-10-05T23:56:03.405Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0
2020-10-05T23:56:03.408Z INFO [autodiscover.pod] kubernetes/util.go:114 kubernetes: Using pod name filebeat-deployment-2ztkx and namespace msopena to discover kubernetes node
2020-10-05T23:56:03.448Z INFO [kubernetes] kubernetes/util.go:120 kubernetes: Using node giuv08 discovered by in cluster pod node query {"libbeat.processor": "add_kubernetes_metadata"}
2020-10-05T23:56:03.448Z INFO [autodiscover.pod] kubernetes/util.go:120 kubernetes: Using node giuv08 discovered by in cluster pod node query
2020-10-05T23:56:03.448Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager
2020-10-05T23:56:04.397Z INFO log/input.go:157 Configured paths: [/var/log/containers/*-20845e0aa57813e0796823e28ddaf92b6bd9d0e84d35e3affec7dd492043d10c.log]
2020-10-05T23:56:04.398Z WARN [cfgwarn] redis/input.go:52 EXPERIMENTAL: Redis slowlog input is enabled.
2020-10-05T23:56:04.401Z INFO log/input.go:157 Configured paths: [/var/log/containers/*-20845e0aa57813e0796823e28ddaf92b6bd9d0e84d35e3affec7dd492043d10c.log]
2020-10-05T23:56:04.402Z WARN [cfgwarn] redis/input.go:52 EXPERIMENTAL: Redis slowlog input is enabled.
2020-10-05T23:56:04.402Z INFO eslegclient/connection.go:99 elasticsearch url: http://148.187.106.204:32341
2020-10-05T23:56:04.407Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.9.1
2020-10-05T23:56:04.412Z ERROR harvester/registry.go:98 Error running input: error receiving slowlog data: dial tcp 148.187.106.204:32527: connect: connection refused
2020-10-05T23:56:06.347Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-10-05T23:56:33.364Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":155}},"total":{"ticks":590,"time":{"ms":601},"value":590},"user":{"ticks":440,"time":{"ms":446}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":13},"info":{"ephemeral_id":"f2b050c5-f027-4ed1-91ce-269946ac578e","uptime":{"ms":30213}},"memstats":{"gc_next":21227840,"memory_alloc":15680656,"memory_total":49206008,"rss":69787648},"runtime":{"goroutines":53}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1}},"output":{"type":"elasticsearch"},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":72},"load":{"1":11.03,"15":9.56,"5":10.1,"norm":{"1":0.1532,"15":0.1328,"5":0.1403}}}}}}
The only events I can see in the Filebeat log are the ones from monitoring the host (Redis events are only shown when I stop and start the container).
My question is:
Is this the expected behaviour or is Filebeats suppossed to fetch and show in its events the same information available in the Redis pod logs?
thank you