# Filebeat fails to harvest if a file and a symlink to that file is in the same directory **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 11, 2016, 9:14am UTC](https://discuss.elastic.co/t/filebeat-fails-to-harvest-if-a-file-and-a-symlink-to-that-file-is-in-the-same-directory/49743 "2016-05-11T09:14:01Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![treff7es1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/treff7es1/32/9708_2.png) [@treff7es1](https://discuss.elastic.co/u/treff7es1)\ **Post date:** [May 11, 2016, 9:14am UTC](https://discuss.elastic.co/t/filebeat-fails-to-harvest-if-a-file-and-a-symlink-to-that-file-is-in-the-same-directory/49743/1 "2016-05-11T09:14:02Z")
Hi, It seems to me if symlink exists in the directory where filebeat should work on then filebeat won't harvest. Our system is writing logs in a format where every hour we name the logfile with the actual hour in the filename and we have a symlink named with current which always points to the latest file like this: log\_2015-01-01\_00001 log\_2015-01-01\_00002 current -\> log\_2015-01-01\_0002 If I start filebeat and symlink is in the actual directory then filebeat fails to harvest any of the files. I could reproduce this issue in the following way: 1. Create a directory 2. Put a logfile there (test.log for example) 3. Create a symlink to that logfile at the same directory (test.current for example) 4. Start filebeat and it won't harvest that file. I tried it with filebeat-5.0.0-alpha2-x86\_64 on Ubuntu 12.04. I could not even find these files in the registry as well: cat data/registry |grep test|wc -l 0 See my debug logs attached. If I set an exclude on the symlink then everything works fine. Here are the debug logs: 2016/05/11 08:56:37.311071 beat.go:276: INFO filebeat start running. 2016/05/11 08:56:37.311147 registrar.go:60: INFO Registry file set to: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/data/registry 2016/05/11 08:56:37.311214 registrar.go:70: INFO Loading registrar data from /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/data/registry 2016/05/11 08:56:37.311665 log.go:12: WARN client/brokers registered new broker #1003 at x.x.x.x 2016/05/11 08:56:37.311781 log.go:12: WARN client/brokers registered new broker #1002 at y.y.y.y 2016/05/11 08:56:37.311945 log.go:16: WARN kafka message: Successfully initialized new client 2016/05/11 08:56:37.312345 spooler.go:41: DBG Spooler will use the default spool\_size of 2048 2016/05/11 08:56:37.312403 spooler.go:47: DBG Spooler will use the default idle\_timeout of 5s 2016/05/11 08:56:37.312531 crawler.go:37: INFO Loading Prospectors: 1 2016/05/11 08:56:37.312576 crawler.go:42: DBG File Configs: [/opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/_] 2016/05/11 08:56:37.312625 prospector.go:241: INFO Set ignore\_older duration to 0 2016/05/11 08:56:37.312666 prospector.go:241: INFO Set scan\_frequency duration to 10s 2016/05/11 08:56:37.312723 prospector.go:177: INFO buffer\_size set to: 16384 2016/05/11 08:56:37.312791 prospector.go:190: INFO input\_type set to: log 2016/05/11 08:56:37.312827 prospector.go:241: INFO Set backoff duration to 1s 2016/05/11 08:56:37.312856 prospector.go:201: INFO backoff\_factor set to: 2 2016/05/11 08:56:37.312886 prospector.go:241: INFO Set max\_backoff duration to 10s 2016/05/11 08:56:37.312932 prospector.go:211: INFO force\_close\_file is disabled 2016/05/11 08:56:37.312964 prospector.go:241: INFO Set close\_older duration to 1h0m0s 2016/05/11 08:56:37.312993 prospector.go:222: INFO max\_bytes set to: 10485760 2016/05/11 08:56:37.313075 prospector\_log.go:40: DBG exclude\_files: [.gz$] 2016/05/11 08:56:37.313116 prospector\_log.go:79: DBG scan path /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/_ 2016/05/11 08:56:37.313281 prospector\_log.go:92: DBG Check file for harvesting: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.current 2016/05/11 08:56:37.313295 spooler.go:75: INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s 2016/05/11 08:56:37.313522 prospector\_log.go:150: DBG Start harvesting unknown file: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.current 2016/05/11 08:56:37.313674 registrar.go:174: INFO New file. Start reading from the beginning: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.current 2016/05/11 08:56:37.313729 prospector\_log.go:263: DBG Start / resuming harvester of file: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.current 2016/05/11 08:56:37.313784 prospector\_log.go:92: DBG Check file for harvesting: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.log 2016/05/11 08:56:37.313837 prospector\_log.go:150: DBG Start harvesting unknown file: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.log 2016/05/11 08:56:37.313711 registrar.go:77: INFO Starting Registrar 2016/05/11 08:56:37.313951 registrar.go:174: INFO New file. Start reading from the beginning: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.log 2016/05/11 08:56:37.314127 prospector\_log.go:251: DBG Launching harvester on renamed file. File rename was detected: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.current -\> /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.log 2016/05/11 08:56:37.314217 log.go:227: DBG harvest: "/opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.current" (offset snapshot:0) 2016/05/11 08:56:37.314287 log.go:41: INFO Harvester started for file: /opt/filebeat/filebeat-5.0.0-alpha2-x86\_64/test/test.current 2016/05/11 08:56:42.313444 spooler.go:132: DBG Flushing spooler because of timeout. Events flushed: 0 Thanks, Tamas ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 12, 2016, 6:14am UTC](https://discuss.elastic.co/t/filebeat-fails-to-harvest-if-a-file-and-a-symlink-to-that-file-is-in-the-same-directory/49743/2 "2016-05-12T06:14:33Z")
@treff7es1 Thanks for the report. I will have a closer look at this. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 20, 2016, 10:52am UTC](https://discuss.elastic.co/t/filebeat-fails-to-harvest-if-a-file-and-a-symlink-to-that-file-is-in-the-same-directory/49743/3 "2016-05-20T10:52:19Z")
@treff7es1 I started to investigate this issue. The first thing I asked myself is what do we expect if there is a symlink to a file which is also in the same directory? Does it just add the orginal to our harvester list? Should we even "respect" symlinks or ignore them? I couldn't reproduce the behaviour about not reading at all, but it currently reads the file twice (which is somehow expected as it treats the symlink like a file). Each symlink has its own inode ... ---
**Author:** ![treff7es1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/treff7es1/32/9708_2.png) [@treff7es1](https://discuss.elastic.co/u/treff7es1)\ **Post date:** [May 20, 2016, 11:24am UTC](https://discuss.elastic.co/t/filebeat-fails-to-harvest-if-a-file-and-a-symlink-to-that-file-is-in-the-same-directory/49743/4 "2016-05-20T11:24:04Z")
@ruflin I would expect to harvest only the file where the symlink points and not add the symlink itself. I think that would be a great idea if it could be set if symlink should be skipped. I will try the usecase again on my side to double check the not harvesting case. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 20, 2016, 11:35am UTC](https://discuss.elastic.co/t/filebeat-fails-to-harvest-if-a-file-and-a-symlink-to-that-file-is-in-the-same-directory/49743/5 "2016-05-20T11:35:08Z")
@treff7es1 I opened a Github issue here with this: [https://github.com/elastic/beats/issues/1686](https://github.com/elastic/beats/issues/1686) Lets move our conversation there. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:51pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-harvest-if-a-file-and-a-symlink-to-that-file-is-in-the-same-directory/49743/6 "2017-07-05T21:51:42Z")