Ignore_older option in filebeat not working as expected

Elastic Stack Beats
1

I just started using filebeat. I have set ignore_older options to 5s. There are a lot of old files in the directory which gets scanned (which cannot be deleted). I see that the old files are opened which causes filebeat to error out since many files are open.

Isin't the ignore_older option supposed to ignore files that were modified before the mentioned value (5s in this case) and thereby not open the files at all?

How else can I achieve not opening old files at all? How can we tell the harvester to not open the old files?

2

I think the problem here is, that to get the meta information about the file (modification time), Golang probably also has to open the file, but should close it directly afterwards again. Can you provide the error messages you get, your OS and filebeat version?

An other upcoming option (release 1.1) you can use is the exclude_files filter, like this the files should never be read. Not sure if that is applicable in your case.

3

Hi Ruflin,
The message i get is that there are many open files. Since the ignore older is set to 5s, every now and then I see some logs being pushed.

This is 1.0.1 on Ubuntu 14.04 64bit

Thanks.

4

How many open files do you allow? About how many log files are we talking?

5

@ruflin: files are created almost every minute and 3 week older files are kept in the location where i am reading from.

Another question - If i want filebeat to send a message to logstash if there was nothing read in the past 5 minutes - how can i achieve that?

6

That means you have in your directory 30k+ files?

For the second question: Filebeat doesn't support this. Can you perhaps elaborate on what your use case and motivation behind this is? Perhaps this leads to an other solution.

7

@ruflin: the use case is the following.

Currently, I have filebeat sending logs to logstash. I have integrated pagerduty with logstash and so when I receive any message with Error, it triggers a page.

I want to enhance this further such that if there is no log received for past 5 minutes, a page should be triggered. I can find time stamps in logstash but unless logstash receives something from filebeat it does not go into the filter section where I have the logic. So, I was thinking if filebeat can send a message to logstash saying it has not received any input in last 5 minutes. Then I can use that input in logstash to trigger the page.

I am open to other solutions.

Thanks for your help!