# Filebeat fails to pickup other files **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 27, 2016, 9:59pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202 "2016-05-27T21:59:03Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [May 27, 2016, 9:59pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/1 "2016-05-27T21:59:03Z")
Hello I run into this type of issue: 2016-05-27T20:05:19Z INFO Old file with new name found: /flow\_base\_dir/dump/flows-201605270835.dump is no /flow\_base\_dir/dump/flows-201605272000.dump 2016-05-27T20:05:19Z INFO Detected rename of a previously harvested file: /flow\_base\_dir/dump/flows-201605270835.dump -\> /flow\_base\_dir/dump/flows-201605272000.dump 2016-05-27T20:05:19Z INFO Registry file updated. 672 states written. 2016-05-27T20:05:29Z INFO Run prospector 2016-05-27T20:05:39Z INFO Run prospector 2016-05-27T20:05:49Z INFO Run prospector 2016-05-27T20:05:59Z INFO Run prospector 2016-05-27T20:06:06Z INFO Read line error: file inactive these dump files are generated every five minutes and they are unique. So filebeat will pick them up and send them to Kafka. I have a cron that will run every 4 hours and delete older files. I wonder if it contributes in any way. Is there to tell filebeat to drop older files, or I guess update registry? ---
**Author:** ![Jaren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaren/32/9986_2.png) [@Jaren](https://discuss.elastic.co/u/Jaren)\ **Post date:** [May 27, 2016, 10:02pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/2 "2016-05-27T22:02:32Z")
to be clear you are saying .. new files are be dropped into the path and are not being processed by filebeat? are the filenames unique? Can you share you config file too? ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [May 27, 2016, 10:28pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/3 "2016-05-27T22:28:26Z")
Yes new files are dropped into the path. Here is snippet. Filenames are unique filebeat: prospectors: - paths: - /flow\_base\_dir/dump/\*.dump document\_type: netflow logging: level: info ``` # enable file rotation with default configuration to_files: true # do not log to syslog to_syslog: false files: path: /home/t/filebeat-5.0.0-alpha1-x86_64/var/log name: filebeat.log keepfiles: 7 ``` ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [May 27, 2016, 10:40pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/4 "2016-05-27T22:40:08Z")
After I restart filebeat things start working ---
**Author:** ![Jaren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaren/32/9986_2.png) [@Jaren](https://discuss.elastic.co/u/Jaren)\ **Post date:** [May 29, 2016, 1:48am UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/5 "2016-05-29T01:48:01Z")
hmmm .... hmmm what are the logs saying? Is it recognizing the new files but not processing them? question about the below log message: are the files unique but are generated multiple times ... i.e. does jaren..log come at 1pm .. get deleted and jaren.log get created again at 2 pm ? `2016-05-27T20:05:19Z INFO Detected rename of a previously harvested file: /flow_base_dir/dump/flows-201605270835.dump -> /flow_base_dir/dump/flows-201605272000.dump` ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 30, 2016, 8:45am UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/6 "2016-05-30T08:45:16Z")
Could it be that you hit this issue here? [https://github.com/elastic/beats/issues/1341](https://github.com/elastic/beats/issues/1341) ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [May 30, 2016, 5:19pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/7 "2016-05-30T17:19:06Z")
Files are unique, since file formatted based on the date/time stamp I added ignore\_older: 15 min, in which case filebeat is picking up files, that i wouldn't do before, but now I am starting to see these: 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 27] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 3] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 12] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 6] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 21] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 18] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 15] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 30] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 9] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 0] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[0 %!d(string=herd-netflow-wc1) 24] state change to [open] on %!s(MISSING)/%!d(MISSING) 2016-05-28T06:45:24Z WARN producer/broker/[1] starting up In this case there is nothing being written to Kafka. Is this something with filebeat or should I check with Kafka. Things seem to pickup byitself ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [May 30, 2016, 5:20pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/8 "2016-05-30T17:20:00Z")
Ignore\_older, supposed to address the inode reuse ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 31, 2016, 8:52am UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/9 "2016-05-31T08:52:19Z")
The warnings above seem to be related to Kafka output (@steffens ? ). About ignore\_older: ignore\_older currently does not solve the inode reuse issue. For more details on options how we plan to solve it, see this issue here: [https://github.com/elastic/beats/issues/1600](https://github.com/elastic/beats/issues/1600) ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 31, 2016, 10:18am UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/10 "2016-05-31T10:18:16Z")
which filebeat version have you running? The kafka messages are by kafka circuit-breaker due to output being unresponsive =\> no lines being published. ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [May 31, 2016, 6:55pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/11 "2016-05-31T18:55:15Z")
I am using filebeat-5.0.0-alpha1-x86\_64, and I meant close\_older not ignore\_older. Is close\_older also not functional? So how can I cleanup these dump files without affecting filebeat? Does that mean I have to restart filebeat every time, I do this clean up? I clean up every 4 hours. Will updating .filebeat registry by removing records that have association to removed files suffice for filebeat to release inodes? If it doesn't require restart, then I can probably go this route. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 1, 2016, 4:45am UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/12 "2016-06-01T04:45:42Z")
I think there is some misunderstand of close\_older and inode reuse as they are not directly related. If something works with a restart, it should also work with keeping filebeat running. Your issue is that new files are not picked up before you restart? Can you please update to alpha3 and check if your problems persist? ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [June 1, 2016, 6:46pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/13 "2016-06-01T18:46:19Z")
So as suggested I upgraded to alpha3. I stopped the cron of having to restart filebeat. Let's see how it does 2016-06-01T18:35:25Z INFO Home path: [/home/t/filebeat-5.0.0-alpha3-linux-x64] Config path: [/home/t/filebeat-5.0.0-alpha3-linux-x64] Data path: [/home/t/filebeat-5.0.0-alpha3-linux-x64/data] Logs path: [/home/t/filebeat-5.0.0-alpha3-linux-x64/logs] 2016-06-01T18:35:25Z INFO Setup Beat: filebeat; Version: 5.0.0-alpha3 2016-06-01T18:35:25Z INFO Activated kafka as output plugin. 2016-06-01T18:35:25Z INFO Publisher name: [blah.com](http://blah.com) 2016-06-01T18:35:25Z INFO Flush Interval set to: 1s 2016-06-01T18:35:25Z INFO Max Bulk Size set to: 2048 2016-06-01T18:35:25Z INFO filebeat start running. 2016-06-01T18:35:25Z INFO Registry file set to: /home/t/filebeat-5.0.0-alpha3-linux-x64/data/registry 2016-06-01T18:35:25Z INFO Loading registrar data from /home/t/filebeat-5.0.0-alpha3-linux-x64/data/registry 2016-06-01T18:35:25Z INFO States Loaded from registrar: 37 2016-06-01T18:35:25Z INFO Loading Prospectors: 1 2016-06-01T18:35:25Z INFO buffer\_size set to: 16384 2016-06-01T18:35:25Z INFO document\_type set to: netflow 2016-06-01T18:35:25Z INFO Invalid input type set: 2016-06-01T18:35:25Z INFO input\_type set to: log 2016-06-01T18:35:25Z INFO Set backoff duration to 1s 2016-06-01T18:35:25Z INFO backoff\_factor set to: 2 2016-06-01T18:35:25Z INFO Set max\_backoff duration to 10s 2016-06-01T18:35:25Z INFO force\_close\_file is disabled 2016-06-01T18:35:25Z INFO Set close\_older duration to 1h0m0s 2016-06-01T18:35:25Z INFO max\_bytes set to: 10485760 2016-06-01T18:35:25Z INFO Load previous states from registry into memory 2016-06-01T18:35:25Z INFO Previous states loaded: 37 2016-06-01T18:35:25Z INFO Loading Prospectors completed. Number of prospectors: 1 2016-06-01T18:35:25Z INFO All prospectors are initialised and running with 37 states to persist 2016-06-01T18:35:25Z INFO Start sending events to output 2016-06-01T18:35:25Z INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s 2016-06-01T18:35:25Z INFO Starting Registrar 2016-06-01T18:35:25Z INFO Starting prospector of type: log 2016-06-01T18:35:25Z INFO Harvester started for file: /flow\_base\_dir/dump/flows-201606011810.dump 2016-06-01T18:35:25Z INFO Harvester started for file: /flow\_base\_dir/dump/flows-201606011745.dump 2016-06-01T18:35:25Z INFO Harvester started for file: /flow\_base\_dir/dump/flows-201606011800.dump 2016-06-01T18:35:25Z INFO Harvester started for file: /flow\_base\_dir/dump/flows-201606011530.dump 2016-06-01T18:35:25Z INFO Harvester started for file: /flow\_base\_dir/dump/flows-201606011600.dump 2016-06-01T18:35:25Z INFO Harvester started for file: /flow\_base\_dir/dump/flows-201606011605.dump ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 2, 2016, 7:08am UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/14 "2016-06-02T07:08:43Z")
Thanks. Let me know in case you hit some issues. ---
**Author:** ![kalapakim](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kalapakim](https://discuss.elastic.co/u/kalapakim)\ **Post date:** [June 2, 2016, 3:27pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/15 "2016-06-02T15:27:04Z")
i'm having the exact issue, if I restart the service it will grab the new logs, but it won't grab any more that are dropped afterwards. But if you run it in command prompt it will pick up all new logs... filebeat -c filebeat filebeat.yml -e I am using alpha 3 as well, was there a specific configuration you had to set to make this work? ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [June 2, 2016, 11:04pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/16 "2016-06-02T23:04:30Z")
I must say, that alpha3 version hasn't had any issues yet. Going to keep my fingers crossed. No issues with old files that have deleted, nor issues with new files being picked up. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 3, 2016, 7:05am UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/17 "2016-06-03T07:05:50Z")
@kalapakim Can you share some more details on your OS? ---
**Author:** ![kalapakim](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kalapakim](https://discuss.elastic.co/u/kalapakim)\ **Post date:** [June 3, 2016, 1:14pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/18 "2016-06-03T13:14:54Z")
Windows Server 2012 and I'm using Filebeat 5.0.0-alpha3. There are more details in the issue I opened... [filebeat-not-polling-the-directory](https://discuss.elastic.co/t/filebeat-not-polling-the-directory/51472) ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [June 7, 2016, 5:17pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/19 "2016-06-07T17:17:25Z")
Just to update, So I am not seeing the rename, or Old file with new name in the filebeat logs anymore. Looks like alpha3 install fixed the issues I was having. I compared the record count in my dump files and what's in ElasticSearch count matches up. So I am not loosing any data. The only thing I am seeing is: Read line error: file inactive This should be due to my old file cleanup. Does this mean that filebeat has released the inode for that file. Does it update filebeat registry automatically? ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [June 15, 2016, 5:31pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202/20 "2016-06-15T17:31:25Z")
Here is another issue I am seeing: I am seeing files being truncated, and I am using filebeat-5.0.0-alpha3-linux-x64 2016-06-15T00:05:20Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150000.dump 2016-06-15T00:10:20Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150005.dump 2016-06-15T00:15:20Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150010.dump 2016-06-15T00:20:21Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150015.dump 2016-06-15T00:25:21Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150020.dump 2016-06-15T00:30:21Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150025.dump 2016-06-15T00:35:11Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150030.dump 2016-06-15T00:40:12Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150035.dump 2016-06-15T00:45:12Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150040.dump 2016-06-15T00:50:12Z INFO File was truncated. Begin reading file from offset 0: /flow\_base\_dir/dump/flows-201606150045.dump I also noticed, that filebeat-5.0.0 alpha4 has been posted, should I test that one? [Next page](https://discuss.elastic.co/t/filebeat-fails-to-pickup-other-files/51202.md?page=2)