Filebeat intermittent auto discovery states issue

Elastic Stack Beats
1

I just switched from fluentd to filebeat and am seeing the following issues for some logs now.
Does anyone have an idea where the issue could reside?

2020-08-03T14:26:23.349Z        ERROR   [autodiscover]  autodiscover/autodiscover.go:210        Auto discover config check failed for config '{
  "docker-json": {
    "cri_flags": true,
    "format": "auto",
    "partial": true,
    "stream": "all"
  },
  "exclude_lines": [
    "^\\s+[\\-`('.|_]"
  ],
  "ignore_older": "48h",
  "multiline": {
    "match": "after",
    "negate": false,
    "pattern": "^[[:space:]]"
  },
  "paths": [
    "/var/log/containers/*-c5913198fe1bcd4fdbf6eca0d777e7d97b3b76ac938d2a0d24122e3af0195011.log"
  ],
  "symlinks": true,
  "type": "container"
}', won't start runner: Can only start an input when all related states are finished: {Id:67529863-2049 Finished:false Fileinfo:0xc004650270 Source:/var/log/containers/grafana-postgres-db-0_sensu-system_postgres-c5913198fe1bcd4fdbf6eca0d777e7d97b3b76ac938d2a0d24122e3af0195011.log Offset:885005 Timestamp:2020-08-03 14:26:18.604698192 +0000 UTC m=+2628.868409771 TTL:-1ns Type:container Meta:map[] FileStateOS:67529863-2049}
2

I've got the exact same issue. After this error message appears, no logs will be collected for the affected container anymore, until the collecting filebeat instance is restarted.

I'm running filebeat-7.8.1 with elasticsearch-7.8.1

3

I think this can help : https://github.com/elastic/beats/issues/11834

4

Update to v7.9.0 fixed the issue for me.
kudos to the elastic-team