Filebeat pod unable to harvest logs even after restart

[bhavaniprasad_reddy]

Filebeat pod unable to harvest logs even after restart.

filebeat.yml

  filebeat.registry_file: /var/tmp/filebeat/filebeat_registry
  filebeat.autodiscover:
    providers:
      - type: kubernetes
        tags:
          - "kube-logs"
        templates:
          - condition:
              or:
                - contains:
                    kubernetes.pod.name: "ne-mgmt"
                - contains:
                    kubernetes.pod.name: "list-manager"
                - contains:
                    kubernetes.pod.name: "scheduler-mgmt"
                - contains:
                    kubernetes.pod.name: "sync-ne"
                - contains:
                    kubernetes.pod.name: "file-manager"
                - contains:
                    kubernetes.pod.name: "dash-board"
                - contains:
                    kubernetes.pod.name: "ne-db-manager"
                - contains:
                    kubernetes.pod.name: "config-manager"
                - contains:
                    kubernetes.pod.name: "report-manager"
                - contains:
                    kubernetes.pod.name: "clean-backup"
                - contains:
                    kubernetes.pod.name: "warrior"
                - contains:
                    kubernetes.pod.name: "ne-backup"
                - contains:
                    kubernetes.pod.name: "ne-restore"
            config:
              - type: docker
                containers.ids:
                  - "${data.kubernetes.container.id}"
                multiline.pattern: '^[[:space:]]'
                multiline.negate: false
                multiline.match: after
  logging.level: debug
  processors:
    - drop_event:
        when.or:
           - equals:
               kubernetes.namespace: "kube-system"
           - equals:
               kubernetes.namespace: "default"
           - equals:
               kubernetes.namespace: "logging"
  output.logstash:
    hosts: ["logstash-service.logging:5044"]
    index: filebeat
    pretty: true
  setup.template.name: "filebeat"
  setup.template.pattern: "filebeat-*"
kibana.yml: |
 elasticsearch.url: "http://index.${NAMESPACE}:9200"

Stack -

filebeat - 6.4.1
logstash - 6.3.1
elasticsearch - 6.5.4
kibana - 6.5.4

Platform - OKD-3.11
Cloud Provider - Openstack
OS - Centos 7
master nodes - 1
worker nodes - 4

Observations:

Filebeat pod running on a particular node is unable to harvest log files even after restart.
Actually there are 300 + pods that are allocated to a node where this filebeat pod is running.
Checked the resources available in the node. The CPU and memory resources are also good.
So deleted the completed pods manually then the pod count came to around 50.
This time filebeat started harvesting the log files after the restart.

There is no clue why filebeat started working after deleting the pods. Please let me know if there is any limitation on the number of logs that filebeat will harvest.