Hi,
I am not sure if this is the best place to discuss this but one of our compute nodes in Kubernetes is constantly alerting on high CPU. After digging a bit deeper, we see that the Filebeat process on the node is triggering the alert. Here are some configuration and metrics:
Filebeat version: 7.0.1
Filebeat configuration with sensitive information redacted:
image:
repository: docker.elastic.co/beats/filebeat-oss
tag: 7.3.0
pullPolicy: IfNotPresent
config:
logging.json: true
filebeat.config:
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/log/messages
- /var/log/syslog
- type: docker
json.keys_under_root: false
json.add_error_key: true
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
- drop_event:
when:
equals:
kubernetes.container.name: "filebeat"
# Disable automatic ILM, since we already manage it separately
setup:
ilm:
enabled: false
template:
name: "<template_name>"
pattern: "<pattern>"
output:
elasticsearch:
hosts: ["<elasticsearch_endpoint>"]
username: "<username>"
# password: "set me through --set config.output.elasticsearch.password=XYZ"
enable: true
bulk_max_size: 1000
index: "<index>"
indices:
- index: "<index>"
path: "/"
output.file:
enabled: false
http.enabled: false
monitoring:
enabled: false
We've not made any changes to Filebeat itself recently. Additionally, the Filebeat logs do not hint at the underlying issue. What else can we look at to figure out a root case for the high CPU alert/s? As an additional debugging step, I did look at the following documents but am not sure if they are relevant:
- https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-cpu.html#:~:text=Filebeat%20is%20using%20too%20much%20CPUedit,disk%20in%20a%20tight%20loop
- Filebeat is using too much cpu
- Filebeat Consumes High CPU:
- Filebeat CPU usage too high
I am happy to provide additional details as needed so any help would be greatly appreciated.