I am currently running ECK 8.6.1 on a bare metal cluster, but I'm experiencing some issues with Filebeats. Depending on the configuration, it either leaks RAM or utilizes an excessive amount of CPU. Here is the configuration file I'm using:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: filebeat
spec:
type: filebeat
version: 8.6.1
elasticsearchRef:
name: elasticsearch
config:
filebeat:
autodiscover:
providers:
- type: kubernetes
add_resource_metadata:
cronjob: false
deployment: false
node: ${NODE_NAME}
templates:
- condition.or:
- equals:
kubernetes.namespace: "namespace1"
- equals:
kubernetes.namespace: "namespace2"
config:
- type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
index: "filebeat-${data.kubernetes.namespace}-%{+yyyy.MM.dd}"
exclude_files:
- ".*rabbitmq.*"
- ".*portal.*"
- ".*businesscard.*"
- ".*airflow.*"
- ".*redis.*"
- ".*mongo.*"
- ".*frontend-app.*"
- ".*rook.*"
- ".*filbeat.*"
- ".*statistics.*"
processors:
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 3
target: ""
overwrite_keys: true
add_error_key: true
expand_keys: true
daemonSet:
podTemplate:
spec:
serviceAccountName: filebeat
automountServiceAccountToken: true
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true # Allows to provide richer host metadata
containers:
- name: filebeat
securityContext:
runAsUser: 0
resources:
limits:
memory: 800Mi
cpu: 2000m
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
With this configuration, CPU utilization goes through the roof, the metrics for the Filebeat daemonset are as follows:
When I remove 'add_resource_metadata.deployment: false' from the configuration, while keeping 'cronjob: false', the CPU utilization returns to normal. However, in this case, Filebeats begins to leak RAM and eventually gets OOMKilled within a few days.
As I am verynew to ECK, I may be overlooking something, and my configuration is definitely flawed. Any assistance in resolving this issue would be greatly appreciated!
