Running Filebeat in docker with persistent registry file?

Elastic Stack Beats
1

Hi, My question is this,

If a container running filebeat is lost and we launch a new container, the registry file of the old container will be lost too and the new container wouldn't know from where the harvester should read the new files which will cause inconsistent/ambiguous data in elasticsearch. It will read the whole file again. So, how should I go with creating a consistent registry file?

Should I also mount /var/lib/filebeat/ to some persistent volume?

Right now, I am thinking in kubernetes(AWS EKS) context. I have a persistent volume(AWS EFS). I have created a pvc(persistent volume claim) for each application, lets say app1-pvc for app1. This pvc is mounted to /var/log/app1/ location both on application and filebeat container. Filebeat is reading input from the file /var/log/app1/api_info.log. So, if my filebeat container gets restarted, the registry file will be lost and the whole api_info will be read again.

Am I thinking in the right way? If yes, then what are the ways to avoid the scenario which I'm facing here? If no, how does it prevent the above scenario? Or where I'm thinking wrong?

2

Hi Luvpreet

You can mount the data directory to a persistent location to enable filebeat to resume reading the files. I have tested this on docker swarm, but you can extend this to k8 as well.

Here is my swarm config:

version: '3.7'
services:
  beats:
    image: "myregistry/filebeat/7.2.0"
    volumes:
      - /filebeat/conf/filebeat.yml:/etc/filebeat/filebeat.yml:ro
      - /filebeat/volumes/data:/usr/share/filebeat/bin/data:rw
    command: "/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml"

The volume and command could change depending on your image or os. Hope this helps.

Cheers!

2 Likes
3

Thanks for the response @NerdSec

My data directory is /usr/share/filebeat/data,

sh-4.2$ pwd
/usr/share/filebeat/data
sh-4.2$ ls
meta.json  registry

So, I guess it will be fine if I only make /usr/share/filebeat/data directory a persistent directory?

4

Cool, I tried it and it worked. I only mounted the data directory.

5

Yes. That would be sufficient. :slight_smile:

Also, in case you wish to delete the old files, have a look here:

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.