Filebeat not showing container logs

Hi,

I have a Kubernetes test cluster with Filebeat running as Daemonset and a Redis container I want to monitor.
Filebeat log shows it can find the container because the logs path (/var/log/containers/*-20845e0aa57813e0796823e28ddaf92b6bd9d0e84d35e3affec7dd492043d10c.log) contains the container ID.

This is my Filebeats configuration:

filebeat.autodiscover:
  providers:
    - type: kubernetes
        templates:
          - condition:
              equals:
                kubernetes.container.image: "redis"
            config:
              - module: redis
                  log:
                    input:
                      type: container
                      paths:
                        - /var/log/containers/*-${data.kubernetes.container.id}.log
                  slowlog:
                    enabled: true
                    var.hosts: ["148.187.106.204:32527"]
processors:
  - add_cloud_metadata: 
  - add_kubernetes_metadata: 
  - add_docker_metadata: 
output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:148.187.106.204}:${ELASTICSEARCH_PORT:32341}']

These are Filebeats logs:

2020-10-05T23:56:03.244Z	INFO	instance/beat.go:640	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-10-05T23:56:03.346Z	INFO	instance/beat.go:648	Beat ID: 23ffb203-c72a-43bc-b5b8-0e35a079366b
2020-10-05T23:56:03.349Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-10-05T23:56:03.349Z	INFO	[beat]	instance/beat.go:976	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "23ffb203-c72a-43bc-b5b8-0e35a079366b"}}}
2020-10-05T23:56:03.349Z	INFO	[beat]	instance/beat.go:985	Build info	{"system_info": {"build": {"commit": "ad823eca4cc74439d1a44351c596c12ab51054f5", "libbeat": "7.9.1", "time": "2020-09-01T19:58:51.000Z", "version": "7.9.1"}}}
2020-10-05T23:56:03.349Z	INFO	[beat]	instance/beat.go:988	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":72,"version":"go1.14.7"}}}
2020-10-05T23:56:03.353Z	INFO	[beat]	instance/beat.go:992	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-31T13:52:36Z","containerized":true,"name":"filebeat-deployment-2ztkx","ip":["127.0.0.1/8","::1/128","10.233.64.27/18","fe80::f5:88ff:fe30:fa88/64"],"kernel_version":"5.4.0-42-generic","mac":["02:f5:88:30:fa:88"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-10-05T23:56:03.354Z	INFO	[beat]	instance/beat.go:1021	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-10-05T23:56:02.530Z"}}}
2020-10-05T23:56:03.354Z	INFO	instance/beat.go:299	Setup Beat: filebeat; Version: 7.9.1
2020-10-05T23:56:03.354Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'filebeat-7.9.1' as ILM is enabled.
2020-10-05T23:56:03.354Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://148.187.106.204:32341
2020-10-05T23:56:03.356Z	INFO	[publisher]	pipeline/module.go:113	Beat name: filebeat-deployment-2ztkx
2020-10-05T23:56:03.360Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-10-05T23:56:03.360Z	INFO	instance/beat.go:450	filebeat start running.
2020-10-05T23:56:03.361Z	INFO	add_kubernetes_metadata/kubernetes.go:71	add_kubernetes_metadata: kubernetes env detected, with version: v1.18.5
2020-10-05T23:56:03.361Z	INFO	[kubernetes]	kubernetes/util.go:114	kubernetes: Using pod name filebeat-deployment-2ztkx and namespace msopena to discover kubernetes node	{"libbeat.processor": "add_kubernetes_metadata"}
2020-10-05T23:56:03.405Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2020-10-05T23:56:03.405Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2020-10-05T23:56:03.405Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2020-10-05T23:56:03.405Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 0
2020-10-05T23:56:03.405Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2020-10-05T23:56:03.408Z	INFO	[autodiscover.pod]	kubernetes/util.go:114	kubernetes: Using pod name filebeat-deployment-2ztkx and namespace msopena to discover kubernetes node
2020-10-05T23:56:03.448Z	INFO	[kubernetes]	kubernetes/util.go:120	kubernetes: Using node giuv08 discovered by in cluster pod node query	{"libbeat.processor": "add_kubernetes_metadata"}
2020-10-05T23:56:03.448Z	INFO	[autodiscover.pod]	kubernetes/util.go:120	kubernetes: Using node giuv08 discovered by in cluster pod node query
2020-10-05T23:56:03.448Z	INFO	[autodiscover]	autodiscover/autodiscover.go:113	Starting autodiscover manager
2020-10-05T23:56:04.397Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*-20845e0aa57813e0796823e28ddaf92b6bd9d0e84d35e3affec7dd492043d10c.log]
2020-10-05T23:56:04.398Z	WARN	[cfgwarn]	redis/input.go:52	EXPERIMENTAL: Redis slowlog input is enabled.
2020-10-05T23:56:04.401Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*-20845e0aa57813e0796823e28ddaf92b6bd9d0e84d35e3affec7dd492043d10c.log]
2020-10-05T23:56:04.402Z	WARN	[cfgwarn]	redis/input.go:52	EXPERIMENTAL: Redis slowlog input is enabled.
2020-10-05T23:56:04.402Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://148.187.106.204:32341
2020-10-05T23:56:04.407Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.9.1
2020-10-05T23:56:04.412Z	ERROR	harvester/registry.go:98	Error running input: error receiving slowlog data: dial tcp 148.187.106.204:32527: connect: connection refused
2020-10-05T23:56:06.347Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-10-05T23:56:33.364Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":155}},"total":{"ticks":590,"time":{"ms":601},"value":590},"user":{"ticks":440,"time":{"ms":446}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":13},"info":{"ephemeral_id":"f2b050c5-f027-4ed1-91ce-269946ac578e","uptime":{"ms":30213}},"memstats":{"gc_next":21227840,"memory_alloc":15680656,"memory_total":49206008,"rss":69787648},"runtime":{"goroutines":53}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1}},"output":{"type":"elasticsearch"},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":72},"load":{"1":11.03,"15":9.56,"5":10.1,"norm":{"1":0.1532,"15":0.1328,"5":0.1403}}}}}}

The only events I can see in the Filebeat log are the ones from monitoring the host (Redis events are only shown when I stop and start the container).

My question is:
Is this the expected behaviour or is Filebeats suppossed to fetch and show in its events the same information available in the Redis pod logs?

thank you

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.