# New files detected but data not sent. offset remains at 0 **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 27, 2016, 12:25am UTC](https://discuss.elastic.co/t/new-files-detected-but-data-not-sent-offset-remains-at-0/56460 "2016-07-27T00:25:45Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![seventy-7](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@seventy-7](https://discuss.elastic.co/u/seventy-7)\ **Post date:** [July 27, 2016, 12:25am UTC](https://discuss.elastic.co/t/new-files-detected-but-data-not-sent-offset-remains-at-0/56460/1 "2016-07-27T00:25:45Z")
Hello Beats community. I have recently migrated from logstash-forwarder to filebeat-1.2.3-1.x86\_64. Running on CentOS 7. I have new application files generated each day which have the date in the filename. There is also a symlink which is rolled to the new days file each day when the application restarts. Filebeats is sending to a logstash receiver. I have noticed across many services, that the file is detected in the registry file but no data is sent and the offset value remains at 0. This is inconsistent and am unable to reproduce always. The issue is resolved after 1, sometimes 2 restarts of filebeats This is the registry file. In this file, the symlink is "pts.au.log" -\> "pts.au-Jul-27-2016.log". You will notice the Jul-27 log is at offset 0. `{ "/home/name/logs/pts.au-Jul-25-2016.log":{ "source":"/home/name/logs/pts.au-Jul-25-2016.log", "offset":371681801, "FileStateOS":{ "inode":539847435, "device":2306 } }, "/home/name/logs/pts.au-Jul-26-2016.log":{ "source":"/home/name/logs/pts.au-Jul-26-2016.log", "offset":366036880, "FileStateOS":{ "inode":539348628, "device":2306 } }, "/home/name/logs/pts.au-Jul-27-2016.log":{ "source":"/home/name/logs/pts.au-Jul-27-2016.log", "offset":0, "FileStateOS":{ "inode":539348630, "device":2306 } }, "/home/name/logs/pts.au.log":{ "source":"/home/name/logs/pts.au.log", "offset":125679360, "FileStateOS":{ "inode":539348630, "device":2306 } }, }` Any ideas why the data is not being sent on the new file? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [July 27, 2016, 8:31am UTC](https://discuss.elastic.co/t/new-files-detected-but-data-not-sent-offset-remains-at-0/56460/2 "2016-07-27T08:31:46Z")
Can you share your filebeat config? Be aware that in 5.0 we removed the reading of symlinks as it caused some duplicate issues. See [https://github.com/elastic/beats/issues/1686](https://github.com/elastic/beats/issues/1686) ---
**Author:** ![seventy-7](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@seventy-7](https://discuss.elastic.co/u/seventy-7)\ **Post date:** [July 27, 2016, 8:59am UTC](https://discuss.elastic.co/t/new-files-detected-but-data-not-sent-offset-remains-at-0/56460/3 "2016-07-27T08:59:40Z")
```auto filebeat: prospectors: - paths: - /home/name/logs/*.log input_type: log document_type: prodlogs scan_frequency: 5s multiline: pattern: '^[[:space:]]+|^Caused by:' negate: false match: after max_lines: 5000 - paths: - /home/name/logs/*.json input_type: log document_type: prodjson scan_frequency: 5s spool_size: 4028 registry_file: /home/name/run/.filebeat output: logstash: hosts: ["logstash:12340", "logstash:12341"] worker: 2 loadbalance: false shipper: logging: to_files: true files: path: /home/name/logs/filebeat name: mybeat keepfiles: 7 ``` Sorry, couldn't keep the indenting here.. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [July 27, 2016, 2:14pm UTC](https://discuss.elastic.co/t/new-files-detected-but-data-not-sent-offset-remains-at-0/56460/4 "2016-07-27T14:14:47Z")
Are the symlink and the file it points to in the same directory? If yes, this will cause issues as the same file is read twice. ---
**Author:** ![seventy-7](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@seventy-7](https://discuss.elastic.co/u/seventy-7)\ **Post date:** [July 29, 2016, 2:40am UTC](https://discuss.elastic.co/t/new-files-detected-but-data-not-sent-offset-remains-at-0/56460/5 "2016-07-29T02:40:41Z")
Yes, they are in the same directory. Thanks for the update... Is there a Issue fix for this which I can track? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [August 2, 2016, 7:50am UTC](https://discuss.elastic.co/t/new-files-detected-but-data-not-sent-offset-remains-at-0/56460/6 "2016-08-02T07:50:57Z")
Can you please open a new issue as a feature request? We closed [https://github.com/elastic/beats/issues/1686](https://github.com/elastic/beats/issues/1686) as we were not sure if this feature is needed. ---
**Author:** ![seventy-7](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@seventy-7](https://discuss.elastic.co/u/seventy-7)\ **Post date:** [August 3, 2016, 11:35pm UTC](https://discuss.elastic.co/t/new-files-detected-but-data-not-sent-offset-remains-at-0/56460/7 "2016-08-03T23:35:25Z")
I wont open a new issue for this. I have a workaround in place: Change the file path from - /home/name/logs/\*.log to - /home/name/logs/_Aug_.log to avoid picking up the symlink. This pull request shall resolve the issue once released: [https://github.com/elastic/beats/pull/1767](https://github.com/elastic/beats/pull/1767) Thanks @ruflin ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 17, 2016, 12:25am UTC](https://discuss.elastic.co/t/new-files-detected-but-data-not-sent-offset-remains-at-0/56460/8 "2016-08-17T00:25:59Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.