I'm running Filebeat on several docker containers using this Dockerfile: https://github.com/primait/docker-filebeat/blob/master/1.2/Dockerfile
When the container for some reason crashes or goes down I need to make sure that the processed logs up to that point are not reprocessed again, leading to a big chunk of duplicated logs on the ELK. I need to avoid this.
From this page directory-layout I get that the data folder (path.data) is where filebeat keeps the processed logs registered. Am I right?
My filebeat.yml has this entry:
But when I enter the docker container and navigate to /etc/filebeat/data I see an empty folder and I'm 100% sure Filebeat is processing logs because I see them live on Kibana.
Can someone help me out here?
Basically my intention is to prevent process the same log entries multiple times in case the docker container crashes.