We use a Filebeat DaemonSet to collect container logs from nodes in our Kubernetes cluster. I recently added some Ubuntu 24.04 nodes to the cluster, and when Filebeat restarts after a reboot generates around 150 MB/s of disk reads, which persists until the pod is killed. During this time, is slow to respond and may not even accept ssh logins. This does not happen on older Ubuntu 22.04 nodes that have identical Filebeat configuration.
The problem was first observed with Filebeat 9.3, and upgrading to Filebeat 9.4 didn't resolve the issue.
The configuration is derived from the example manifest at GitHub, which apparently I am not allowed to link to. These are the relevant sections:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
# filebeat.inputs:
# - type: filestream
# id: kubernetes-container-logs
# paths:
# - /var/log/containers/*.log
# parsers:
# - container: ~
# prospector:
# scanner:
# fingerprint.enabled: true
# symlinks: true
# file_identity.fingerprint: ~
# processors:
# - add_kubernetes_metadata:
# host: ${NODE_NAME}
# matchers:
# - logs_path:
# logs_path: "/var/log/containers/"
# Customization: Enable hints based autodiscover
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: filestream
id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
parsers:
- container: ~
prospector:
scanner:
fingerprint.enabled: true
symlinks: true
file_identity.fingerprint: ~
processors:
- add_cloud_metadata:
- add_host_metadata:
# Customization: Comment out unused Elastic Cloud config
#cloud.id: ${ELASTIC_CLOUD_ID}
#cloud.auth: ${ELASTIC_CLOUD_AUTH}
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
# allow connecting to older versions of Elasticsearch to decouple upgrades
allow_older_versions: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
# Customization: Add tolerations to collect control plane logs
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
- effect: NoSchedule
key: node-role.kubernetes.io/master
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat-wolfi:9.4.4
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: "REDACTED_HOSTNAME"
- name: ELASTICSEARCH_PORT
value: "9500"
- name: ELASTICSEARCH_USERNAME
valueFrom:
secretKeyRef:
name: elasticsearch-creds
key: filebeat_user
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: elasticsearch-creds
key: filebeat_password
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0640
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
# When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
path: /var/lib/filebeat-data
type: DirectoryOrCreate
The nodes are lightly loaded (15-20 pods), and none of the logs are particularly large (megabytes, not gigabytes). And although iotop reports that Filebeat is doing a large amount of reads, not that many files are open.
$ sudo lsof -p 47892
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
filebeat 47892 root cwd DIR 0,73 4096 63065921 /usr/share/filebeat
filebeat 47892 root rtd DIR 0,73 4096 63045881 /
filebeat 47892 root txt REG 0,73 255987936 63066226 /usr/share/filebeat/filebeat
filebeat 47892 root mem REG 0,73 63065979 /usr/lib/libnss_compat.so.2 (stat: No such file or directory)
filebeat 47892 root mem REG 0,73 63065972 /usr/lib/libc.so.6 (stat: No such file or directory)
filebeat 47892 root mem REG 0,73 63065982 /usr/lib/libpthread.so.0 (stat: No such file or directory)
filebeat 47892 root mem REG 0,73 63065974 /usr/lib/libdl.so.2 (stat: No such file or directory)
filebeat 47892 root mem REG 0,73 63065983 /usr/lib/libresolv.so.2 (stat: No such file or directory)
filebeat 47892 root mem REG 0,73 63065969 /usr/lib/ld-linux-x86-64.so.2 (stat: No such file or directory)
filebeat 47892 root 0u CHR 1,3 0t0 6 /dev/null
filebeat 47892 root 1w FIFO 0,14 0t0 416444 pipe
filebeat 47892 root 2w FIFO 0,14 0t0 416445 pipe
filebeat 47892 root 3r REG 0,30 0 11268 /sys/fs/cgroup/cpu.max
filebeat 47892 root 4r REG 252,1 7014 61210685 /var/log/pods/default_lgh-batch-prod-29756161-bh6rp_2e370314-cef1-4d27-a72e-7b0c1d3bb50e/lgh-batch-prod/0.log
filebeat 47892 root 5u a_inode 0,15 0 56 [eventpoll:6,17]
filebeat 47892 root 6u a_inode 0,15 0 56 [eventfd:32]
filebeat 47892 root 7r REG 252,1 7017 61210684 /var/log/pods/default_lgh-batch-stage-29756161-49hz8_06fc41d9-7f19-487d-942c-23c96749ba52/lgh-batch-stage/0.log
filebeat 47892 root 9r REG 252,1 5852595 61210681 /var/log/pods/kube-system_filebeat-8kncb_7ffdeda6-5819-4ca5-9870-35e81df8bbf8/filebeat/0.log
filebeat 47892 root 11rW REG 252,1 0 61735277 /usr/share/filebeat/data/filebeat.lock
filebeat 47892 root 12r REG 252,1 8016 61210636 /var/log/pods/monitoring_kps-prometheus-node-exporter-b8b2w_42b1dcc3-c9b0-4bcd-88e7-d01f881fbb2c/node-exporter/1.log
filebeat 47892 root 13r REG 252,1 10687 61210690 /var/log/pods/kube-flannel_kube-flannel-ds-6zj89_506e77bc-744c-45e7-8d6e-62de59ab4cb2/kube-flannel/1.log
filebeat 47892 root 14u REG 252,1 1344573 61746391 /usr/share/filebeat/data/registry/filebeat/log.json
filebeat 47892 root 15r REG 252,1 4066 61079859 /var/log/pods/kube-system_kube-proxy-dx6wd_054b2508-e533-4347-98a6-4d88a6f42e06/kube-proxy/5.log
filebeat 47892 root 17u sock 0,8 0t0 1865676 protocol: TCP
I have looked through the GitHub issues and haven't found anything obviously relevant. Any suggestions for what I should do here?