Filebeat Pod restarting continuously due to OOM

Elastic Stack Beats
,
1

Hi All,

We have deployed Filebeat as a pod in our environment.
We have set memory limit of 2GB in the deployment resources. We still see the pods getting restarted almost every day.
Attaching the configmap and deployment files used to create the pod.

Filebeat pod status
Filebeat pod status1251×530 17.8 KB
Filebeat restarts
Filebeat restarts829×62 3.86 KB

Filebeat deployment:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "7"
  creationTimestamp: "2020-08-21T11:31:53Z"
  generation: 7
  labels:
    app: filebeat
  name: filebeat-deployment-2
  namespace: cloud-prod
  resourceVersion: "61963424"
  selfLink: /apis/apps/v1/namespaces/cloud-prod/deployments/filebeat-deployment-2
  uid: 825ca49f-1f12-4380-b8e7-92b1b8162424
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: filebeat
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: filebeat
    spec:
      containers:
      - image: docker.elastic.co/beats/filebeat:6.1.4
        imagePullPolicy: IfNotPresent
        name: filebeat
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /mnt/api-logs
          name: logs-pv2
        - mountPath: /usr/share/filebeat/filebeat.yml
          name: filebeat-prod-cm-2
          subPath: filebeat.yml
        - mountPath: /tmp
          name: filebeat-certs
        - mountPath: /usr/share/filebeat/data
          name: filebeat-data-pvc-1
      dnsPolicy: ClusterFirst
      hostAliases:
      - hostnames:
        - logstash
        ip: 169.48.213.132
      initContainers:
      - command:
        - chown
        - -R
        - 1000:1000
        - /var/www
        image: busybox
        imagePullPolicy: Always
        name: chown
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/www
          name: filebeat-data-pvc-1
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: logs-pv2
        persistentVolumeClaim:
          claimName: prod-pv-claim-dal12
      - name: filebeat-data-pvc-1
        persistentVolumeClaim:
          claimName: filebeat-data-pvc-1
      - configMap:
          defaultMode: 420
          name: filebeat-prod-cm-2
        name: filebeat-prod-cm-2
      - name: filebeat-certs
        secret:
          defaultMode: 420
          secretName: filebeat-certs
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-09-03T10:17:38Z"
    lastUpdateTime: "2020-11-24T12:42:32Z"
    message: ReplicaSet "filebeat-deployment-2-544dbb6844" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-12-15T02:34:26Z"
    lastUpdateTime: "2020-12-15T02:34:26Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 7
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Filebeat configmap:

Please edit the object below. Lines beginning with a '#' will be ignored,

and an empty file will abort the edit. If an error occurs while saving this file will be

reopened with the relevant failures.

apiVersion: v1
data:
filebeat.yml: |
filebeat.prospectors:
- type: log
paths:
- /mnt/api-logs/*
output.logstash:
hosts: ["logstash:31190"]
ssl.certificate_authorities: ["/tmp/ca.cer"]
ssl.certificate: "/tmp/client.cer"
ssl.key: "/tmp/client.key"
ssl.verification_mode: "none"
kind: ConfigMap
metadata:
creationTimestamp: "2020-08-21T10:56:50Z"
name: filebeat-prod-cm-2
namespace: cloud-prod
resourceVersion: "40098014"
selfLink: /api/v1/namespaces/cloud-prod/configmaps/filebeat-prod-cm-2
uid: c88bf066-7b48-428f-b385-43427aab988b

Please let me know what other details are required to get to the bottom of this issue?

Regards,
Pavan

2

Yeah, chasing memory issues can be tricky. You can take a look at a similar thread: Filebeat high memory usage v6.4.2

In general we don't support filebeat 6.x, so please keep in mind that any optimizations or fixes would rather be available in newer releases.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.