From this comment it seems ignore_older should be bigger the maximum update period, so files are not closed too soon and still picked up.
Can you try to run filebeat with -e -d 'prospector' to log some debug message (and other logs) to stderr?
After closing a file due to ignore_older prospector might generate different debug messages for this file:
- (always if last now - mod_time > ignore_older) Fetching old state of file to resume
- Resuming harvester on a previously harvested file
- Skipping file ...
- (if now - mod_time < ignore_older and file renamed) File rename was detected
- (if now - mod_time < ignore_older and no file rename)
- (if file is already known) Resuming harvester on a previously harvested file
- (if new file) Launching harvester on new file
The timestamps on your log-file match the actual log event timestamp. Filebeat uses the file modification time. If you know a file its name being affected in advance you can try this line of bash to get unique mod-time updates at about every second:
$ logfile=...; while true; do sleep 1; ls -l --full-time $logfile; done | uniq