Filebeat on Kubernetes EKS "file info is not identical with opened file. Aborting harvesting "

[Babadofar (Christoffer Vig)]

In a kubernetes cluster on Amazon EKS with heavy logging we find that there is some missing data from elasticsearch. Log rotation is set up on a 10MB schedule, which means log rotation sometimes happens every minute.
Not sure what is happening, but logs like the below, from filebeat, could perhaps be an indication?
Any idea on how to setup filebeat to handle this?
We are using the standard AWS EKS docker-daemon.json https://github.com/awslabs/amazon-eks-ami/blob/master/files/docker-daemon.json

2019-10-21T10:01:06.665Z	INFO	log/harvester.go:280	File is inactive: /var/lib/docker/containers/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2-json.log. Closing because close_inactive of 5m0s reached.
2019-10-21T10:03:43.428Z	INFO	log/harvester.go:255	Harvester started for file: /var/lib/docker/containers/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2-json.log
2019-10-21T10:09:37.803Z	ERROR	log/input.go:484	Harvester could not be started on new file: /var/lib/docker/containers/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2-json.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: file info is not identical with opened file. Aborting harvesting and retrying file later again

[pmercado (PM)]

Hi @Babadofar,

can you post here your configuration and filebeat version?

[Babadofar (Christoffer Vig)]

filebeat 6.7.1, using mostly defaults, this is the input type docker config:

 - type: docker
  multiline.pattern: ^[[:space:]]
  multiline.negate: false
  multiline.match: after
  containers.ids:
  - "*"
  processors:
    - add_kubernetes_metadata:
        in_cluster: true

[pmercado (PM)]

can you check if the file that filebeat is not able to open exists?
in such case, can you also check if there might be some permissions issue?

[Babadofar (Christoffer Vig)]

This is in prod, so a little bit hard to check out.
Suspect it needs more CPU and/or memory.
There is also a datadog agent reading the same log files.