Inactive and Rolling files


I have a question regarding the close_inactive status and rolling files configuration of Filebeat.

I currently have Filebeat shipping some apache logs.
Apache Log entries can come in every 2 seconds as well as every 48 hours (This is a test environment that is not heavily used).

There is a rolling mechanism gzipping access_log file every day.

Currently the Filebeat configuration is using the default 5m close_inactive settings.

- type: log
    - "/var/log/httpd/access_log"
      application: apache2
      type: access
  fields_under_root: true

So when I start the beat it ships apache logs to logstash.
After 5m of Apache inactivity on the access_log I get
File is inactive: /var/log/httpd/access_log. Closing because close_inactive of 5m0s reached
and Filebeat stops sending log updates to logstash after that.

New logs arriving in access_log are not shipped anymore.
So after 5 minutes of inactivity of Apache I am left with a Filebeat closing the file and not shipping logs thereafter.

If a file is updated after the harvester is closed, the file will be picked up again after scan_frequency has elapsed.

If you require log lines to be sent in near real time do not use a very low scan_frequency but adjust close_inactive so the file handler stays open and constantly polls your files.
The default setting is 10s.

I understood that scan_frequency would be executed by default every 10s making Filebeat monitoring access_log new entries even after the file has been closed for inactivity.
This is actually not happening. After close_inactive time has passed new logs are never picked up again.

The other doubt I have is related to Filebeat file handing when a file gets rolled (gzipped).
Is there something I need to set in order to have Filebeat keep track of the new /var/log/httpd/access_log once the rolling operation is triggered?
It seems that when the file is gzipped I am also loosing Filebeat track of it.


scan_frequency seems to behave correctly on my local box running Docker Engine Community 18.09.0
I start thinking it might be a problem with the Docker version installed on the server 1.13.1

Docker version is the only difference I see (or maybe something at docker config level).
'mick' user is actually part of docker group on both systems

I have noticed this

mick@local:/tmp $ docker run -v /tmp:/tmp alpine touch /tmp/test.txt
mick@local:/tmp $ ls -lart
-rw-r--r--  1 mick    mick     0B Dec 20 23:50 test.txt

while on the Server the test.txt results own by root

mick@server:/tmp $ docker run -v /tmp:/tmp alpine touch /tmp/test.txt
mick@server:/tmp $ ls -lart
-rw-r--r--  1 root    root     0B Dec 20 23:50 test.txt

Not sure why is this happening.
Could this cause problems with Filebeat scan_frequency configuration?

As I said, on the Server once the File has been closed, Filebeats will never scan it again for updates and it stays at


On the local box if the logs receives new updates it gets reopened.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.