# Filebeat Auto discover config check failed for config - stopped harvesting data

**URL:** <https://discuss.elastic.co/t/filebeat-auto-discover-config-check-failed-for-config-stopped-harvesting-data/253675>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [October 29, 2020, 10:42am UTC](https://discuss.elastic.co/t/filebeat-auto-discover-config-check-failed-for-config-stopped-harvesting-data/253675 "2020-10-29T10:42:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dorinand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorinand/32/70521_2.png) [@dorinand](https://discuss.elastic.co/u/dorinand)\
**Post date:** [October 29, 2020, 10:42am UTC](https://discuss.elastic.co/t/filebeat-auto-discover-config-check-failed-for-config-stopped-harvesting-data/253675/1 "2020-10-29T10:42:13Z")

</div>

I harvest logs from gitlab runners created in kubernetes namespace. I implement the opciton to store elastic index in pod annotation. So all my gitlab runners has `elastic.index: runner-jobs` annotation.

This is my autodsicover configuration:

```
filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log

```

The logs has been harvested properly for a long time, but since I implement annotation, it stopped working. I find out in my logs error about check config failed.

```
2020-10-29T10:19:32.486Z ERROR [autodiscover] autodiscover/autodiscover.go:210 Auto discover config check failed for config '{ │
  "docker-json": { │
    "cri_flags": true, │
    "format": "auto", │
    "partial": true, │
    "stream": "all" │
  }, │
  "paths": [ │
    "/var/log/containers/*9a15c9051d6dfa378c6438b67067cb92b8c29233c8b50ea23b176204fc7ed76d.log" │
  ], │
  "symlinks": true, │
  "type": "container" │
}', won't start runner: Can only start an input when all related states are finished: {Id:9877265-65025 Finished:false Fileinfo:0xc0004a7e10 Source:/var/log/containers/runner-yecxnkr-project-340-concurrent-12f6pw_gitlab-runners_helper-9a15c9051

```

I find [this](https://github.com/elastic/beats/issues/11834) issue on github, but still not sure how to solve this and basically, where is the problem. Could anybody give an explanation whats going on, and how to fix this issue?

I am using filebeat version `7.8.1` deployed from [this](https://github.com/elastic/helm-charts) helm chart.

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [October 29, 2020, 11:23am UTC](https://discuss.elastic.co/t/filebeat-auto-discover-config-check-failed-for-config-stopped-harvesting-data/253675/2 "2020-10-29T11:23:18Z")

</div>

Hey!

You pointed to the correct GH issue. It should be [solved now](https://github.com/elastic/beats/issues/11834#issuecomment-683668819)! Could you try with a newer version like `7.9`?

---

<div class="post-metadata">

**Author:** ![dorinand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorinand/32/70521_2.png) [@dorinand](https://discuss.elastic.co/u/dorinand)\
**Post date:** [October 30, 2020, 2:18pm UTC](https://discuss.elastic.co/t/filebeat-auto-discover-config-check-failed-for-config-stopped-harvesting-data/253675/3 "2020-10-30T14:18:18Z")

</div>

Thank you for your reply. I will check it once I will move to new version (not so soon I think).

Do I understand correctly from [this](https://github.com/elastic/beats/issues/11834#issuecomment-511770898) comment:

> When this error message appears it means, that autodiscover attempted to create new Input but in registry it was not marked as finished (probably some other input is reading this file). Autodiscover then attempts to retry creating input every 10 seconds. So if you keep getting error every 10s you have probably something misconfigured. Otherwise you should be fine.

That means, this error is ok because it will start harversting logs later, in the next round of discover process? Or is it necessary to move to newer version of ELK?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [October 30, 2020, 2:51pm UTC](https://discuss.elastic.co/t/filebeat-auto-discover-config-check-failed-for-config-stopped-harvesting-data/253675/4 "2020-10-30T14:51:19Z")

</div>

The issue was making Filebeat to stop shipping logs for an updated pod. `7.8` was hitting the issue, and [https://github.com/elastic/beats/pull/20305](https://github.com/elastic/beats/pull/20305) was backported to `7.8.x` but we never had a `7.8.2` release so you will need `7.9` to solve this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 27, 2020, 4:51pm UTC](https://discuss.elastic.co/t/filebeat-auto-discover-config-check-failed-for-config-stopped-harvesting-data/253675/5 "2020-11-27T16:51:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.