We have configured logstash with file input to scan files from specified directory and index into elastic search. We have provided the following path for file input : /var/log/varnish/varnish-access*.log
Varnish generates log files for every hour.
We have changed the stat_interval for file input for 1 hr.
stat_interval => 3600
rest all parameters are by default.
Our observation is that when we set stat_interval to 1Hr the files there is a lag between the ingestion. i.e. files earlier than 6 Hrs are getting scanned. We checked with the lsof command so only the files earlier than 5-6 Hrs are opened by logstash. even if we set the stat_interval to 1Hr.
Is there any delay in the scanning (discovering the new files) and processing if we set stat_interval to higher value ?
Also we tried to change stat_interval to 1 min then the logstash was able to pickup the files faster.
Is there any relation between the stat_interval, since_db_write interval ?