Filebeats is re-processing logs once it restarts

[aniketkk (Aniket Kulkarni)]

Hi, I am using Nginx as my app server. I have installed filebeat where Nginx is installed.
Filebeat pushes these logs to Logstash. Then logstash pushes forward to elasticsearch.

But whenever I do a deployment the previous log records which were already processed is sent to logstash again and there is a creation of duplicate entries.

I can think the of problem which might be causing it and have mentioned it below

We have dockerized the filebeat and whenever the new deployment goes, it fetches the latest image and is restarted. But the logs are stored on instance running the container (using docker volumes). So the old log files always remain. But the registry files are lost when the docker is restarted. So it loses the offset and it again starts harvesting from the beginning of the file. So maybe that is the reason for it to reprocess older logs. Correct me if I am wrong.

Now regarding the solution.

So is using ignore_older directive the solution? If yes, what should be the suitable value for it. If we set ignore_older, is it mandatory to set clean_inactive. Please suggest me the best-suited configuration for my case (an nginx server)

I was thinking about one more solution. Why how about exposing /usr/share/filebeat/data directory which contains registry files to the host instance running the container. In this way, even if the filebeat container restarts during the deployment, the registry files would remain in the host instance like the logs. So it would pick from the correct offset and send only relevant logs.

I am thinking second the solution perhaps would be better. Please suggest me the best solution.

Regards

[steffens (Steffen Siering)]

Filebeat is stateful and stores it's state in the registry file. You can reconfigure the path of the registry file. But in essence, the registry file must be available when you restart filebeat in a new container. Use either volumes or host-mount to share /usr/share/filebeat/data.

[aniketkk (Aniket Kulkarni)]

Thanks @steffens. I tried to do host mount. Ran my custom filebeat image with volume as -v /usr/share/filebeat/data:/usr/share/filebeat/data. But now the container is not able to create registry file at the location because of permission issue.

My dockerfile looks like this

FROM docker.elastic.co/beats/filebeat:6.2.2
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chown filebeat /usr/share/filebeat/filebeat.yml
USER filebeat 

Here as recommended in the docs, I run as user filebeat. But the host mounted directory is having root permissions. Hence my container is failing to write registry files.

What is the solution for this? Or Can I run filebeat as root?

[ruflin]

You can run filebeat as root if you want. Alternative would be to set the right permissions on your mounted container.

[aniketkk (Aniket Kulkarni)]

Thanks, @ruflin. Well if I go the changing permissions on the mounted volume route, there are around 15-20's of servers in the cloud where I need to make this change. I was wondering if I do something in the Dockerfile that reflects in any of the servers without changing it in the start-up script of each n every server. And I also wanted to avoid running as root.

A solution to somewhat is a similar problem is answered here.

How to give non-root user in Docker container access to a volume mounted on the host

I was wondering if we could do something like this. I don't know the side effects of using CMD or ENTRYPOINT in my Dockerfile because it would override the CMD of the beats docker file and may end up having some side effects.

What do you think of this type of solution?

[exekias (Carlos Pérez Aradros)]

Hi @aniketkk,

We use this as entrypoint: https://github.com/elastic/beats-docker/blob/master/templates/Dockerfile.j2#L49

You can call that from your custom script, and it should work Just take into account that if this changes at some point in our side you will need to update yours.

Best regards