Inactive and Rolling files

[mickt (mick)]

Hi,

I have a question regarding the close_inactive status and rolling files configuration of Filebeat.

I currently have Filebeat shipping some apache logs.
Apache Log entries can come in every 2 seconds as well as every 48 hours (This is a test environment that is not heavily used).

There is a rolling mechanism gzipping access_log file every day.

Currently the Filebeat configuration is using the default 5m close_inactive settings.

filebeat.inputs:
- type: log
  paths:
    - "/var/log/httpd/access_log"
  fields:
    log:
      application: apache2
      type: access
  fields_under_root: true

So when I start the beat it ships apache logs to logstash.
After 5m of Apache inactivity on the access_log I get
File is inactive: /var/log/httpd/access_log. Closing because close_inactive of 5m0s reached
and Filebeat stops sending log updates to logstash after that.

New logs arriving in access_log are not shipped anymore.
So after 5 minutes of inactivity of Apache I am left with a Filebeat closing the file and not shipping logs thereafter.

If a file is updated after the harvester is closed, the file will be picked up again after scan_frequency has elapsed.

scan_frequency
If you require log lines to be sent in near real time do not use a very low scan_frequency but adjust close_inactive so the file handler stays open and constantly polls your files.
The default setting is 10s.

I understood that scan_frequency would be executed by default every 10s making Filebeat monitoring access_log new entries even after the file has been closed for inactivity.
This is actually not happening. After close_inactive time has passed new logs are never picked up again.

The other doubt I have is related to Filebeat file handing when a file gets rolled (gzipped).
Is there something I need to set in order to have Filebeat keep track of the new /var/log/httpd/access_log once the rolling operation is triggered?
It seems that when the file is gzipped I am also loosing Filebeat track of it.

Thanks

[mickt (mick)]

scan_frequency seems to behave correctly on my local box running Docker Engine Community 18.09.0
I start thinking it might be a problem with the Docker version installed on the server 1.13.1

Docker version is the only difference I see (or maybe something at docker config level).
'mick' user is actually part of docker group on both systems

I have noticed this

mick@local:/tmp $ docker run -v /tmp:/tmp alpine touch /tmp/test.txt
mick@local:/tmp $ ls -lart
-rw-r--r--  1 mick    mick     0B Dec 20 23:50 test.txt

while on the Server the test.txt results own by root

mick@server:/tmp $ docker run -v /tmp:/tmp alpine touch /tmp/test.txt
mick@server:/tmp $ ls -lart
-rw-r--r--  1 root    root     0B Dec 20 23:50 test.txt

Not sure why is this happening.
Could this cause problems with Filebeat scan_frequency configuration?

As I said, on the Server once the File has been closed, Filebeats will never scan it again for updates and it stays at

"filebeat":{"harvester":{"open_files":0,"running":0}}

On the local box if the logs receives new updates it gets reopened.