Hi,
I have a cluster running Kubernetes version 1.24, hosted on AWS EKS.
Into the cluster there are multiple workloads segregated by namespace.
Filebeat is installed to run as a DaemonSet, and should be able to collect logs for a subset of k8s namespaces.
Till Kubernetes 1.23 all was working fine, but after the removal of Dockershim in Kubernetes 1.24, the old configuration isn't working anymore.
I have tried many different configurations, to collect logs based on containerd:
filebeat:
bulk_max_size: 8000
filebeat.inputs:
- type: filestream
id: obs-filebeat
paths:
- /var/log/pods/**/*.log
prospector.scanner.exclude_files: ['filebeat.*',
'logstash.*',
'kube.*',
'cert-manager.*',
'kube-proxy-*',
'fluentd-*',
'cloudwatch-agent-*',
'obs-filebeat-*',
'lens-metrics_node-*']
processors:
- add_kubernetes_metadata:
in_cluster: true
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: '/var/log/pods/'
default_indexers.enabled: false
default_matchers.enabled: false
indexers:
- pod_uid:
matchers:
- logs_path:
logs_path: '/var/log/pods/'
resource_type: 'pod'
- drop_event.when.not.and:
- equals.kubernetes.namespace: "my-app"
processors:
- add_fields:
when:
equals:
kubernetes.namespace: "my-app"
target: ''
fields:
env: "prod"
app: "my-app"
- decode_json_fields:
fields: ["message","log","logs.log","parsed-message"]
target: "logs"
process_array: true
- drop_fields:
when:
regexp:
message: "^{\""
fields: ["message"]
ignore_missing: true
- drop_fields:
fields: ["kubernetes.node", "log.file.path","timestamp","std","capital-letter","parsed-message"]
ignore_missing: true
output.logstash:
enabled: true
hosts: ["logstash:5044"]
bulk_max_size: 8000
workers: 8
loadbalance: true
pipelining: 3
compression_level: 1
timeout: 60s
max_retries: -1
template.name: filebeat
template.path: filebeat.template.json
logging.to_files: false
queue.disk:
path: "${path.data}/persistent-diskqueue"
max_size: 30GB
When I start Filebeats, logs are collected without any problem and sent to Logstash, but after a while Filebeat seems to stop reading new logs.
{"log.level":"info","@timestamp":"2023-09-05T10:32:52.310Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":8}},"cpuacct":{"total":{"ns":25524289}},"memory":{"mem":{"usage":{"bytes":163725312}}}},"cpu":{"system":{"ticks":1230,"time":{"ms":20}},"total":{"ticks":2220,"time":{"ms":30},"value":2220},"user":{"ticks":990,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"2f6da9c4-2a94-4284-a83b-260249aecef4","uptime":{"ms":360102},"version":"8.8.2"},"memstats":{"gc_next":17970568,"memory_alloc":12250840,"memory_total":162885240,"rss":91398144},"runtime":{"goroutines":27}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":6.33,"15":7.19,"5":7.17,"norm":{"1":1.5825,"15":1.7975,"5":1.7925}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-09-05T10:33:22.309Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":10}},"cpuacct":{"total":{"ns":21054857}},"memory":{"mem":{"usage":{"bytes":163733504}}}},"cpu":{"system":{"ticks":1250,"time":{"ms":20}},"total":{"ticks":2240,"time":{"ms":20},"value":2240},"user":{"ticks":990}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"2f6da9c4-2a94-4284-a83b-260249aecef4","uptime":{"ms":390103},"version":"8.8.2"},"memstats":{"gc_next":17970568,"memory_alloc":13518344,"memory_total":164152744,"rss":91398144},"runtime":{"goroutines":27}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":6.92,"15":7.2,"5":7.22,"norm":{"1":1.73,"15":1.8,"5":1.805}}}},"ecs.version":"1.6.0"}}
Filebeat version: 8.8.2
Any idea?