Filebeat Pods OOMKilled

Elastic Stack Beats
,
1

Hi

Filebeat version: 7.9.3
I am running Filebeat as a deployment on IBM Kubernetes and reading the logs from a PVC mounted on the cluster. I am seeing Filebeat pods restarting due to OOMKilled.
There are some log files in the mounted PVC that do not have the necessary permissions for Filebeat to read and I only see ERROR message for that in the logs.
Filebeat deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2021-09-22T08:14:20Z"
  generation: 5
  labels:
    app: filebeat
  name: filebeat-wdc04
  namespace: prod
  resourceVersion: "33463237"
  uid: 2ecff80d-902b-41ab-b024-2585a888452f
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: filebeat
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: filebeat
    spec:
      containers:
      - image: docker.elastic.co/beats/filebeat:7.9.3
        imagePullPolicy: IfNotPresent
        name: filebeat
        resources:
          limits:
            cpu: "2"
            memory: 1Gi
          requests:
            cpu: 400m
            memory: 400Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /mnt/api-logs/
          name: prod-pv-claim-wdc04
        - mountPath: /usr/share/filebeat/filebeat.yml
          name: filebeat-config-wdc04
          subPath: filebeat.yml
        - mountPath: /usr/share/filebeat/data
          name: filebeat-data-pvc-wdc04
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - chown
        - -R
        - 1000:1000
        - /var/www
        image: busybox
        imagePullPolicy: Always
        name: chown
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/www
          name: filebeat-data-pvc-wdc04
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: prod-pv-claim-wdc04
        persistentVolumeClaim:
          claimName: prod-pv-claim-wdc04
      - name: filebeat-data-pvc-wdc04
        persistentVolumeClaim:
          claimName: filebeat-data-pvc-wdc04
      - configMap:
          defaultMode: 420
          name: filebeat-config-wdc04
        name: filebeat-config-wdc04
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2021-09-22T08:14:21Z"
    lastUpdateTime: "2021-09-22T08:14:35Z"
    message: ReplicaSet "filebeat-wdc04-5ff4f5645f" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2021-11-18T08:23:45Z"
    lastUpdateTime: "2021-11-18T08:23:45Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 5
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Filebeat configmap

apiVersion: v1
data:
  filebeat.yml: |
    filebeat.inputs:
    - type: log
      paths:
       - /mnt/api-logs/*
      tags: ["wdc04","phi02"]
      ignore_older: 2h
    output.logstash:
      hosts: ["169.48.231.149:5044"]
kind: ConfigMap
metadata:
  creationTimestamp: "2021-09-22T08:11:06Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:filebeat.yml: {}
    manager: kubectl
    operation: Update
    time: "2021-09-22T08:11:06Z"
  name: filebeat-config-wdc04
  namespace: prod
  resourceVersion: "7717519"
  uid: 97fa218c-8e54-459a-95ef-a6b831d0c77f

The PVC Access mode is RWX

Is there a way to troubleshoot this issue?

Best Regards,
Pavan

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.