I am trying to start using Filebeat and I have a huge number of log files. I have tried with option tail_files to true but "too many files open" error comes. Then I tried setting ignore_older to 2h and close_older to 30m but Filebeat is not sending logs. What is the best way to initialise Filebeat in such an environment?
What is the number of files you have? Is it log files so most of them are actually not needed anymore? You could try to set the allowed number of open files on your system higher. Second you could try to set ignore_older much lower.
This issue came up once or twice in the past. We started also discussing it in the past and one solution is to limit the max number of harvesters that is started. The challenge here is for which files the harvesters are started first?
There is an other issue you will potentially hit with that many files: The size of the registry file. For more details see here: https://github.com/elastic/beats/issues/1600 We are currently trying to solve this problem.
Thanks a lot for quick response. I already tried by keeping ignore_older to much lower and it works.. But I found some strange issue, I have one Java program which generates a .err file whenever an exception occurs. But when I include this path in filebeat like /var/log/my_comp/*.err, the filebeat seems to be hanging. If I remove this path, everything is working fine. One more thing, most of the time the err files are of 0 size. But I tried adding a dummy file with some content, it is not picked by filebeat. This is happening only for err files of Java component, for another component it is working smoothly. I ran filebeat with only one path and debug mode,
The output was
2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my_comp/20160602132801.err
2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my_comp/20160602132901.err
2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my_comp/20160602133001.err
2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my_comp/20160602133101.err
2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my_comp/20160602133201.err
2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my_comp/20160602133301.err
2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my_comp/20160602133401.err
2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my_comp/20160602133501.err
2016-06-16T14:03:45Z DBG Check file for harvesting: /var/logs/my_comp/20160602133601.err
2016-06-16T14:03:45Z DBG Check file for harvesting: /var/logs/my_comp/20160602133701.err
Do your error files have a new line character at the end of the file/event?
Hi,
Yes, there is a new line added after every content in the err files.
One more thing I also get file inactive error, is it due to this new line character error?
Whenever I add a new path to the .yml, should I change the registry also ?
Normally you never should have to change the registry, new files are automatically added. You only have to remove the registry, if you want to start crawling from scratch again.
What is the exact error you get? What do you mean by the new line character error?
As you said whether I have new line at the end of the file. The answer of that is yes. As I said when those err files are added to the registry the filebeat doesn't work properly, it keeps on looping among the files but doesn't flush to the output.
Can you start filebeat again in debug mode with -e -d "*" and share the output?
The following lines are getting printed in loop. Nothing is flushed. I have added a new .err file but it has not been picked up too..
2016-06-23T09:30:10Z DBG Check file for harvesting: /data/logs/my_comp/20160623073901.err
2016-06-23T09:30:11Z DBG Check file for harvesting: /data/logs/my_comp/20160623074001.err
2016-06-23T09:30:11Z DBG Check file for harvesting: /data/logs/my_comp/20160623074101.err
2016-06-23T09:30:14Z DBG Check file for harvesting: /data/logs/my_comp/20160623074201.err
2016-06-23T09:30:15Z DBG Check file for harvesting: /data/logs/my_comp/20160623074301.err
2016-06-23T09:30:15Z DBG Check file for harvesting: /data/logs/my_comp/20160623074401.err
2016-06-23T09:30:15Z DBG Check file for harvesting: /data/logs/my_comp/20160623074501.err
2016-06-23T09:30:16Z DBG Check file for harvesting: /data/logs/my_comp/20160623074601.err
2016-06-23T09:30:17Z DBG Check file for harvesting: /data/logs/my_comp/20160623074701.err
2016-06-23T09:30:17Z DBG Check file for harvesting: /data/logs/my_comp/20160623074801.err
2016-06-23T09:30:18Z DBG Flushing spooler because of timeout. Events flushed: 0
2016-06-23T09:30:18Z DBG Check file for harvesting: /data/logs/my_comp/20160623074902.err
2016-06-23T09:30:19Z DBG Check file for harvesting: /data/logs/my_comp/20160623075001.err
2016-06-23T09:30:19Z DBG Check file for harvesting: /data/logs/my_comp/20160623075101.err
2016-06-23T09:30:20Z DBG End of file reached: /data/logs/comp_2/plog/20160623092101.log; Backoff now.
2016-06-23T09:30:21Z DBG Check file for harvesting: /data/logs/my_comp/20160623075201.err
2016-06-23T09:30:21Z DBG Check file for harvesting: /data/logs/my_comp/20160623075301.err
2016-06-23T09:30:21Z DBG Check file for harvesting: /data/logs/my_comp/20160623075401.err
2016-06-23T09:30:21Z DBG Check file for harvesting: /data/logs/my_comp/20160623075501.err
2016-06-23T09:30:22Z DBG Check file for harvesting: /data/logs/my_comp/20160623075601.err
2016-06-23T09:30:23Z DBG Check file for harvesting: /data/logs/my_comp/20160623075701.err
2016-06-23T09:30:23Z DBG Check file for harvesting: /data/logs/my_comp/20160623075802.err
2016-06-23T09:30:24Z DBG Check file for harvesting: /data/logs/my_comp/20160623075901.err
2016-06-23T09:30:24Z DBG Check file for harvesting: /data/logs/my_comp/20160623080001.err
2016-06-23T09:30:25Z DBG Check file for harvesting: /data/logs/my_comp/20160623080101.err
2016-06-23T09:30:25Z DBG Flushing spooler because of timeout. Events flushed: 0
Nothing special shows up in the log entries above. It seems to only start harvesters. How many .err files exist in the directory?
This topic was automatically closed after 21 days. New replies are no longer allowed.