# Filebeat on Kubernetes EKS "file info is not identical with opened file. Aborting harvesting " **URL:** \ **Category:** Beats\ **Tags:** docker, filebeat\ **Created:** [October 21, 2019, 11:24am UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-eks-file-info-is-not-identical-with-opened-file-aborting-harvesting/204449 "2019-10-21T11:24:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar)\ **Post date:** [October 21, 2019, 11:24am UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-eks-file-info-is-not-identical-with-opened-file-aborting-harvesting/204449/1 "2019-10-21T11:24:47Z")
In a kubernetes cluster on Amazon EKS with heavy logging we find that there is some missing data from elasticsearch. Log rotation is set up on a 10MB schedule, which means log rotation sometimes happens every minute. Not sure what is happening, but logs like the below, from filebeat, could perhaps be an indication? Any idea on how to setup filebeat to handle this? We are using the standard AWS EKS docker-daemon.json [https://github.com/awslabs/amazon-eks-ami/blob/master/files/docker-daemon.json](https://github.com/awslabs/amazon-eks-ami/blob/master/files/docker-daemon.json) ``` 2019-10-21T10:01:06.665Z INFO log/harvester.go:280 File is inactive: /var/lib/docker/containers/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2-json.log. Closing because close_inactive of 5m0s reached. 2019-10-21T10:03:43.428Z INFO log/harvester.go:255 Harvester started for file: /var/lib/docker/containers/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2-json.log 2019-10-21T10:09:37.803Z ERROR log/input.go:484 Harvester could not be started on new file: /var/lib/docker/containers/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2/5bf73d583f8a15cccf1c6a17050a71f8f44c1d8e22b0ce521ca106f58a68cbf2-json.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: file info is not identical with opened file. Aborting harvesting and retrying file later again ``` ---
**Author:** ![pmercado](https://avatars.discourse-cdn.com/v4/letter/p/59ef9b/32.png) [@pmercado](https://discuss.elastic.co/u/pmercado)\ **Post date:** [October 21, 2019, 11:42am UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-eks-file-info-is-not-identical-with-opened-file-aborting-harvesting/204449/2 "2019-10-21T11:42:23Z")
Hi @Babadofar, can you post here your configuration and filebeat version? ---
**Author:** ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar)\ **Post date:** [October 21, 2019, 12:14pm UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-eks-file-info-is-not-identical-with-opened-file-aborting-harvesting/204449/3 "2019-10-21T12:14:09Z")
filebeat 6.7.1, using mostly defaults, this is the input type docker config: ``` - type: docker multiline.pattern: ^[[:space:]] multiline.negate: false multiline.match: after containers.ids: - "*" processors: - add_kubernetes_metadata: in_cluster: true ``` ---
**Author:** ![pmercado](https://avatars.discourse-cdn.com/v4/letter/p/59ef9b/32.png) [@pmercado](https://discuss.elastic.co/u/pmercado)\ **Post date:** [October 24, 2019, 8:09am UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-eks-file-info-is-not-identical-with-opened-file-aborting-harvesting/204449/4 "2019-10-24T08:09:22Z")
can you check if the file that filebeat is not able to open exists? in such case, can you also check if there might be some permissions issue? ---
**Author:** ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar)\ **Post date:** [October 28, 2019, 11:47am UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-eks-file-info-is-not-identical-with-opened-file-aborting-harvesting/204449/5 "2019-10-28T11:47:16Z")
This is in prod, so a little bit hard to check out. Suspect it needs more CPU and/or memory. There is also a datadog agent reading the same log files. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 25, 2019, 11:47am UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-eks-file-info-is-not-identical-with-opened-file-aborting-harvesting/204449/6 "2019-11-25T11:47:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.