Filebeat Auto discover config check failed for config - stopped harvesting data

[dorinand]

I harvest logs from gitlab runners created in kubernetes namespace. I implement the opciton to store elastic index in pod annotation. So all my gitlab runners has elastic.index: runner-jobs annotation.

This is my autodsicover configuration:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log

The logs has been harvested properly for a long time, but since I implement annotation, it stopped working. I find out in my logs error about check config failed.

2020-10-29T10:19:32.486Z    ERROR    [autodiscover]    autodiscover/autodiscover.go:210    Auto discover config check failed for config '{                                                                                                                                            │
  "docker-json": {                                                                                                                                                                                                                                                                    │
    "cri_flags": true,                                                                                                                                                                                                                                                                │
    "format": "auto",                                                                                                                                                                                                                                                                 │
    "partial": true,                                                                                                                                                                                                                                                                  │
    "stream": "all"                                                                                                                                                                                                                                                                   │
  },                                                                                                                                                                                                                                                                                  │
  "paths": [                                                                                                                                                                                                                                                                          │
    "/var/log/containers/*9a15c9051d6dfa378c6438b67067cb92b8c29233c8b50ea23b176204fc7ed76d.log"                                                                                                                                                                                       │
  ],                                                                                                                                                                                                                                                                                  │
  "symlinks": true,                                                                                                                                                                                                                                                                   │
  "type": "container"                                                                                                                                                                                                                                                                 │
}', won't start runner: Can only start an input when all related states are finished: {Id:9877265-65025 Finished:false Fileinfo:0xc0004a7e10 Source:/var/log/containers/runner-yecxnkr-project-340-concurrent-12f6pw_gitlab-runners_helper-9a15c9051

I find this issue on github, but still not sure how to solve this and basically, where is the problem. Could anybody give an explanation whats going on, and how to fix this issue?

I am using filebeat version 7.8.1 deployed from this helm chart.

[ChrsMark (Chris Mark)]

Hey!

You pointed to the correct GH issue. It should be solved now! Could you try with a newer version like 7.9?

[dorinand]

Thank you for your reply. I will check it once I will move to new version (not so soon I think).

Do I understand correctly from this comment:

When this error message appears it means, that autodiscover attempted to create new Input but in registry it was not marked as finished (probably some other input is reading this file). Autodiscover then attempts to retry creating input every 10 seconds. So if you keep getting error every 10s you have probably something misconfigured. Otherwise you should be fine.

That means, this error is ok because it will start harversting logs later, in the next round of discover process? Or is it necessary to move to newer version of ELK?

[ChrsMark (Chris Mark)]

The issue was making Filebeat to stop shipping logs for an updated pod. 7.8 was hitting the issue, and https://github.com/elastic/beats/pull/20305 was backported to 7.8.x but we never had a 7.8.2 release so you will need 7.9 to solve this.