File input plugin to process files not newer than X seconds ago

Elastic Stack Logstash
1

Hello again.

I am trying to use file input plugin, read mode, delete after completion, but I need to take in charge just files that are not newer than X seconds ago.

This because my logger application writes a new file (to be then took in charge by Logstash) each second and I want to avoid to take in charge a file that is going to be written completely. So, I can be sure it can be deleted.

Is there a way to do so?

2

No, you can ignore files that are older than some specific time, but you cannot configure it to ignore files that are newer than some specific time.

The read mode should be used with files that are already completed, if you have an application still writing into the file, it may have some issues.

I think you may use stat_interval and discover_interval to increase the interval in which Logstash checks for new files in the configured path.

The first setting is related to checking if a file has been modified, it makes more sense when using tail mode, the second setting is a multiplier for the first one and is related to the interval in which logstash checks for new files.

The default values are 1 second and 15, so Logstash checks for new files every 15 seconds.

1 Like