# Harvester not started for new files in configured paths **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 13, 2022, 9:46pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332 "2022-01-13T21:46:00Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![dchsueh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dchsueh/32/100299_2.png) [@dchsueh](https://discuss.elastic.co/u/dchsueh)\ **Post date:** [January 13, 2022, 9:46pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332/1 "2022-01-13T21:46:00Z")
Hello, I have a problem I've verified in filebeat 6.8.3, 7.5.1, and **7.16.3** I have a filebeat.yml filebeat.inputs, type log, with three configured paths with \*\* in the middle portions, something like ```auto - /var/log/a/b/**/ONE - /var/log/a/b/**/TWO - /var/log/a/b/**/THREE ``` and on filebeat startup, in a situation where multiple files pre-exist, e.g. for the "\*\*" portion above, I have C/D, C/E, C/F, e.g. ```auto /var/log/a/b/C/D/ONE /var/log/a/b/C/D/TWO /var/log/a/b/C/D/THREE /var/log/a/b/C/E/ONE /var/log/a/b/C/E/TWO /var/log/a/b/C/E/THREE /var/log/a/b/C/F/ONE /var/log/a/b/C/F/TWO /var/log/a/b/C/F/THREE ``` filebeat will establish an input\_id for it and start harvesters for all present files. However, any NEW files that appear after startup (e.g. a new intermediary directory C/G with its own files ONE TWO THREE) will rarely have a harvester started up for it. Occasionally a single harvester will start up for a single file but this is very rare. If no files exist on startup that match the input\_id's configured paths, then filebeat will properly notice the three files appearing and start harvesters for this. The above behavior confirmed both via filebeat -d 'input' and lsof on the process. My read of the documentation and previous discussions (e.g. [Filebeats not harvesting new file - #2 by pierhugues](https://discuss.elastic.co/t/filebeats-not-harvesting-new-file/163400/2)) suggest that filebeat should always find and harvest new files that match the configured paths. I've tried modifying scan\_frequency to no effect. I'd appreciate any ideas to debug/solve, or confirmation to file a github issue. Thank you. ---
**Author:** ![dchsueh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dchsueh/32/100299_2.png) [@dchsueh](https://discuss.elastic.co/u/dchsueh)\ **Post date:** [January 13, 2022, 10:07pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332/2 "2022-01-13T22:07:36Z")
Restarting filebeat results in harvesters started to collect all files present (incl ones missed by previous process). No change with symlinks:true (where /var/log -\> /mnt/var/log on my system). On 7.16.3, splitting into three separate inputs with one configured path each increases the probability that a harvester will start, but it will not get all new files. IIRC, splitting into separate inputs on 7.5.1 does not improve the situation. ---
**Author:** ![dchsueh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dchsueh/32/100299_2.png) [@dchsueh](https://discuss.elastic.co/u/dchsueh)\ **Post date:** [January 24, 2022, 5:31pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332/3 "2022-01-24T17:31:54Z")
Anybody have any ideas? I realize this works for most people; for me in some environments it seems to work reliably but in others never. ---
**Author:** ![Anandu\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anandu_d/32/98900_2.png) [@Anandu\_D](https://discuss.elastic.co/u/Anandu_D)\ **Post date:** [February 7, 2022, 4:46pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332/4 "2022-02-07T16:46:33Z")
Did you find any solution for this problem ---
**Author:** ![dchsueh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dchsueh/32/100299_2.png) [@dchsueh](https://discuss.elastic.co/u/dchsueh)\ **Post date:** [February 8, 2022, 9:46pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332/5 "2022-02-08T21:46:10Z")
no solution found yet ---
**Author:** ![dchsueh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dchsueh/32/100299_2.png) [@dchsueh](https://discuss.elastic.co/u/dchsueh)\ **Post date:** [March 4, 2022, 10:07pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332/6 "2022-03-04T22:07:47Z")
no solution found yet ---
**Author:** ![dchsueh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dchsueh/32/100299_2.png) [@dchsueh](https://discuss.elastic.co/u/dchsueh)\ **Post date:** [March 22, 2022, 11:02pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332/7 "2022-03-22T23:02:07Z")
no solution found yet ---
**Author:** ![dchsueh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dchsueh/32/100299_2.png) [@dchsueh](https://discuss.elastic.co/u/dchsueh)\ **Post date:** [April 18, 2022, 3:19pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332/8 "2022-04-18T15:19:48Z")
no solution found yet ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 16, 2022, 5:20pm UTC](https://discuss.elastic.co/t/harvester-not-started-for-new-files-in-configured-paths/294332/9 "2022-05-16T17:20:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.