Harvester could not be started on existing file due to registry already stopped

Elastic Stack Beats
,
1

After deploying Elasticsearch, Kibana with FileBeat version 8.1.2 in a Kubernetes Cluster, I've noticed the following error message coming over and over:

Harvester could not be started on existing file: /var/log/containers/domain-risk-web-6ff7f9c87b-bbsqv_mss-prod_domain-risk-web-e3ab7c4557d3cc2a7ac8780e7ed0bcdb8d5c73ba25cac3446edfc10d97a321e9.log, Err: registry already stopped

This error is coming from filebeat/input/log/input.go#L582. It seems like the source of this error is filebeat/harvester/registry.go#L81.

I'm unable to figure out the problem, and how to fix it. It seems like this particular issue is not common after doing some searching in Elastic Discuss and Github Issues. Someone posted a very similar issue (existing file vs new file) on Elastic Discuss, and the post didn't receive any comments.

The error message is not very helpful and there are no comments in the codebase describing the scenarios that could trigger this particular error or any recommendation on how to fix it.

beat.yaml

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 8.1.2
  elasticsearchRef:
    name: elastic
  kibanaRef:
    name: kibana
  config:
    setup.dashboards:
      # See https://www.elastic.co/guide/en/beats/filebeat/current/configuration-dashboards.html
      enabled: true
      retry:
        enabled: true
        interval: 4
        maximum: 8
    filebeat.autodiscover.providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints:
        enabled: true
        default_config:
          type: container
          paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
          - name: filebeatout
            mountPath: /var/log/filebeat
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: filebeatout
          hostPath: 
            path: /var/log/filebeat
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.