# Starting Filebeat with large number of logs **URL:** \ **Category:** Beats\ **Created:** [June 16, 2016, 2:42am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938 "2016-06-16T02:42:39Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![avitash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avitash/32/10346_2.png) [@avitash](https://discuss.elastic.co/u/avitash)\ **Post date:** [June 16, 2016, 2:42am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/1 "2016-06-16T02:42:39Z")
I am trying to start using Filebeat and I have a huge number of log files. I have tried with option tail\_files to true but "too many files open" error comes. Then I tried setting ignore\_older to 2h and close\_older to 30m but Filebeat is not sending logs. What is the best way to initialise Filebeat in such an environment? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 16, 2016, 11:04am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/2 "2016-06-16T11:04:34Z")
What is the number of files you have? Is it log files so most of them are actually not needed anymore? You could try to set the allowed number of open files on your system higher. Second you could try to set ignore\_older much lower. This issue came up once or twice in the past. We started also discussing it in the past and one solution is to limit the max number of harvesters that is started. The challenge here is for which files the harvesters are started first? There is an other issue you will potentially hit with that many files: The size of the registry file. For more details see here: [https://github.com/elastic/beats/issues/1600](https://github.com/elastic/beats/issues/1600) We are currently trying to solve this problem. ---
**Author:** ![avitash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avitash/32/10346_2.png) [@avitash](https://discuss.elastic.co/u/avitash)\ **Post date:** [June 16, 2016, 2:19pm UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/3 "2016-06-16T14:19:17Z")
Thanks a lot for quick response. I already tried by keeping ignore\_older to much lower and it works.. But I found some strange issue, I have one Java program which generates a .err file whenever an exception occurs. But when I include this path in filebeat like /var/log/my\_comp/\*.err, the filebeat seems to be hanging. If I remove this path, everything is working fine. One more thing, most of the time the err files are of 0 size. But I tried adding a dummy file with some content, it is not picked by filebeat. This is happening only for err files of Java component, for another component it is working smoothly. I ran filebeat with only one path and debug mode, The output was 2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my\_comp/20160602132801.err 2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my\_comp/20160602132901.err 2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my\_comp/20160602133001.err 2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my\_comp/20160602133101.err 2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my\_comp/20160602133201.err 2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my\_comp/20160602133301.err 2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my\_comp/20160602133401.err 2016-06-16T14:03:44Z DBG Check file for harvesting: /var/logs/my\_comp/20160602133501.err 2016-06-16T14:03:45Z DBG Check file for harvesting: /var/logs/my\_comp/20160602133601.err 2016-06-16T14:03:45Z DBG Check file for harvesting: /var/logs/my\_comp/20160602133701.err ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 17, 2016, 9:58am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/4 "2016-06-17T09:58:55Z")
Do your error files have a new line character at the end of the file/event? ---
**Author:** ![avitash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avitash/32/10346_2.png) [@avitash](https://discuss.elastic.co/u/avitash)\ **Post date:** [June 17, 2016, 11:01am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/5 "2016-06-17T11:01:10Z")
Hi, Yes, there is a new line added after every content in the err files. One more thing I also get file inactive error, is it due to this new line character error? ---
**Author:** ![avitash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avitash/32/10346_2.png) [@avitash](https://discuss.elastic.co/u/avitash)\ **Post date:** [June 17, 2016, 12:50pm UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/6 "2016-06-17T12:50:59Z")
Whenever I add a new path to the .yml, should I change the registry also ? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 20, 2016, 7:29am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/7 "2016-06-20T07:29:08Z")
Normally you never should have to change the registry, new files are automatically added. You only have to remove the registry, if you want to start crawling from scratch again. What is the exact error you get? What do you mean by the new line character error? ---
**Author:** ![avitash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avitash/32/10346_2.png) [@avitash](https://discuss.elastic.co/u/avitash)\ **Post date:** [June 20, 2016, 5:14pm UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/8 "2016-06-20T17:14:59Z")
As you said whether I have new line at the end of the file. The answer of that is yes. As I said when those err files are added to the registry the filebeat doesn't work properly, it keeps on looping among the files but doesn't flush to the output. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 21, 2016, 11:15am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/9 "2016-06-21T11:15:16Z")
Can you start filebeat again in debug mode with `-e -d "*"` and share the output? ---
**Author:** ![avitash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avitash/32/10346_2.png) [@avitash](https://discuss.elastic.co/u/avitash)\ **Post date:** [June 23, 2016, 9:36am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/10 "2016-06-23T09:36:10Z")
The following lines are getting printed in loop. Nothing is flushed. I have added a new .err file but it has not been picked up too.. 2016-06-23T09:30:10Z DBG Check file for harvesting: /data/logs/my\_comp/20160623073901.err 2016-06-23T09:30:11Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074001.err 2016-06-23T09:30:11Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074101.err 2016-06-23T09:30:14Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074201.err 2016-06-23T09:30:15Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074301.err 2016-06-23T09:30:15Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074401.err 2016-06-23T09:30:15Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074501.err 2016-06-23T09:30:16Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074601.err 2016-06-23T09:30:17Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074701.err 2016-06-23T09:30:17Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074801.err 2016-06-23T09:30:18Z DBG Flushing spooler because of timeout. Events flushed: 0 2016-06-23T09:30:18Z DBG Check file for harvesting: /data/logs/my\_comp/20160623074902.err 2016-06-23T09:30:19Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075001.err 2016-06-23T09:30:19Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075101.err 2016-06-23T09:30:20Z DBG End of file reached: /data/logs/comp\_2/plog/20160623092101.log; Backoff now. 2016-06-23T09:30:21Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075201.err 2016-06-23T09:30:21Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075301.err 2016-06-23T09:30:21Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075401.err 2016-06-23T09:30:21Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075501.err 2016-06-23T09:30:22Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075601.err 2016-06-23T09:30:23Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075701.err 2016-06-23T09:30:23Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075802.err 2016-06-23T09:30:24Z DBG Check file for harvesting: /data/logs/my\_comp/20160623075901.err 2016-06-23T09:30:24Z DBG Check file for harvesting: /data/logs/my\_comp/20160623080001.err 2016-06-23T09:30:25Z DBG Check file for harvesting: /data/logs/my\_comp/20160623080101.err 2016-06-23T09:30:25Z DBG Flushing spooler because of timeout. Events flushed: 0 ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 28, 2016, 10:13am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/11 "2016-06-28T10:13:02Z")
Nothing special shows up in the log entries above. It seems to only start harvesters. How many .err files exist in the directory? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 7, 2016, 2:42am UTC](https://discuss.elastic.co/t/starting-filebeat-with-large-number-of-logs/52938/12 "2016-07-07T02:42:54Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.