Filebeat pods are restarting with "panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod" error

I have deployed Filebeat -> Logstash -> elasticseach -> kibana in OKD-3.11 platform running on Openstack Centos-7.6 VMs. Using v7.10.2 oss docker images for the deployment.

There are 3 filebeat pods running one on each worker node, 1 logstash pod, 1 elasticsearch pod and a kibana pod running on the cluster.

filebeat.yml

  filebeat.autodiscover:
    providers:
      - type: kubernetes
        node: ${NODE_NAME}
        tags:
          - "kube-logs"
        templates:
          - condition:
              or:
                - contains:
                    kubernetes.pod.name: "ne-mgmt"
                - contains:
                    kubernetes.pod.name: "list-manager"
                - contains:
                    kubernetes.pod.name: "scheduler-mgmt"
                - contains:
                    kubernetes.pod.name: "sync-ne"
                - contains:
                    kubernetes.pod.name: "file-manager"
                - contains:
                    kubernetes.pod.name: "dash-board"
                - contains:
                    kubernetes.pod.name: "ne-db-manager"
                - contains:
                    kubernetes.pod.name: "config-manager"
                - contains:
                    kubernetes.pod.name: "report-manager"
                - contains:
                    kubernetes.pod.name: "clean-backup"
                - contains:
                    kubernetes.pod.name: "warrior"
                - contains:
                    kubernetes.pod.name: "ne-backup"
                - contains:
                    kubernetes.pod.name: "ne-restore"
            config:
              - type: container
                paths:
                  - "/var/log/containers/*-${data.kubernetes.container.id}.log"
  logging.level: debug
  processors:
    - drop_event:
        when.or:
           - equals:
               kubernetes.namespace: "kube-system"
           - equals:
               kubernetes.namespace: "default"
           - equals:
               kubernetes.namespace: "logging"
  output.logstash:
    hosts: ["logstash-service.logging:5044"]
    index: filebeat
    pretty: true
  setup.template.name: "filebeat"
  setup.template.pattern: "filebeat-*"

The Filebeat pods are restarting with "cache.DeletedFinalStateUnknown" as mentioned below,

2021-03-06T11:46:39.829Z        DEBUG   [input] input/input.go:139      Run input
2021-03-06T11:46:39.829Z        DEBUG   [input] log/input.go:205        Start next scan
2021-03-06T11:46:39.830Z        DEBUG   [input] log/input.go:439        Check file for harvesting: /var/log/containers/1615031071454-ne-backup-1615031071454-dz52x_task-execution_tes-job-ctr-ffaaen22-1615031073017-572e5c521cc95667c7a75a5903e13b12f5c8c6cdbe18e42005602986eaee8c07.log
2021-03-06T11:46:39.830Z        DEBUG   [input] log/input.go:530        Update existing file for harvesting: /var/log/containers/1615031071454-ne-backup-1615031071454-dz52x_task-execution_tes-job-ctr-ffaaen22-1615031073017-572e5c521cc95667c7a75a5903e13b12f5c8c6cdbe18e42005602986eaee8c07.log, offset: 1115
2021-03-06T11:46:39.830Z        DEBUG   [input] log/input.go:582        Harvester for file is still running: /var/log/containers/1615031071454-ne-backup-1615031071454-dz52x_task-execution_tes-job-ctr-ffaaen22-1615031073017-572e5c521cc95667c7a75a5903e13b12f5c8c6cdbe18e42005602986eaee8c07.log
2021-03-06T11:46:39.830Z        DEBUG   [input] log/input.go:226        input states cleaned up. Before: 1, After: 1, Pending: 0
panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod

goroutine 71680 [running]:
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnDelete.func1()
        /go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:174 +0x7c
created by time.goFunc
        /usr/local/go/src/time/sleep.go:168 +0x44

The filebeat pod logs are extending for more than 1 lakh lines so added only a portion of the log here - filebeat_DeletedFinalStateUnknown - 2e44ff3d

Observations -
Filebeat pods are restarting every time after the "etcd_master" & "api_master" docker containers are exiting.

Platform Details -
Master node - 1
Worker Nodes - 3
Platform - OKD-3.11
OS - Centos-7.6
Kubernetes - 1.11

Resources Allocated to Filebeat -
RAM - 2 to 4 Gb
CPU - 2 to 4 cores

Resources allocated to Logstash -
RAM - 2 to 4 Gb
CPU - 2 to 4 cores

Please let me know if you need any more details and suggest me the steps to resolve this.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.