New files detected but data not sent. offset remains at 0

[seventy-7 (Steven)]

Hello Beats community.

I have recently migrated from logstash-forwarder to filebeat-1.2.3-1.x86_64. Running on CentOS 7. I have new application files generated each day which have the date in the filename. There is also a symlink which is rolled to the new days file each day when the application restarts. Filebeats is sending to a logstash receiver.

I have noticed across many services, that the file is detected in the registry file but no data is sent and the offset value remains at 0. This is inconsistent and am unable to reproduce always. The issue is resolved after 1, sometimes 2 restarts of filebeats

This is the registry file. In this file, the symlink is "pts.au.log" -> "pts.au-Jul-27-2016.log". You will notice the Jul-27 log is at offset 0.

{ "/home/name/logs/pts.au-Jul-25-2016.log":{ "source":"/home/name/logs/pts.au-Jul-25-2016.log", "offset":371681801, "FileStateOS":{ "inode":539847435, "device":2306 } }, "/home/name/logs/pts.au-Jul-26-2016.log":{ "source":"/home/name/logs/pts.au-Jul-26-2016.log", "offset":366036880, "FileStateOS":{ "inode":539348628, "device":2306 } }, "/home/name/logs/pts.au-Jul-27-2016.log":{ "source":"/home/name/logs/pts.au-Jul-27-2016.log", "offset":0, "FileStateOS":{ "inode":539348630, "device":2306 } }, "/home/name/logs/pts.au.log":{ "source":"/home/name/logs/pts.au.log", "offset":125679360, "FileStateOS":{ "inode":539348630, "device":2306 } }, }

Any ideas why the data is not being sent on the new file?

[ruflin]

Can you share your filebeat config? Be aware that in 5.0 we removed the reading of symlinks as it caused some duplicate issues. See https://github.com/elastic/beats/issues/1686

[seventy-7 (Steven)]

filebeat:
  prospectors:
    -
      paths:
        - /home/name/logs/*.log
      input_type: log
      document_type: prodlogs
      scan_frequency: 5s
      multiline:
        pattern: '^[[:space:]]+|^Caused by:'
        negate: false
        match: after
        max_lines: 5000
    -
      paths:
        - /home/name/logs/*.json
      input_type: log
      document_type: prodjson
      scan_frequency: 5s
  spool_size: 4028
  registry_file: /home/name/run/.filebeat
output:
  logstash:
    hosts: ["logstash:12340", "logstash:12341"]
    worker: 2
    loadbalance: false
shipper:
logging:
  to_files: true
  files:
    path: /home/name/logs/filebeat
    name: mybeat
    keepfiles: 7

Sorry, couldn't keep the indenting here..

[ruflin]

Are the symlink and the file it points to in the same directory? If yes, this will cause issues as the same file is read twice.

[seventy-7 (Steven)]

Yes, they are in the same directory.
Thanks for the update...

Is there a Issue fix for this which I can track?

[ruflin]

Can you please open a new issue as a feature request? We closed https://github.com/elastic/beats/issues/1686 as we were not sure if this feature is needed.

[seventy-7 (Steven)]

I wont open a new issue for this. I have a workaround in place:

Change the file path from
- /home/name/logs/*.log
to
- /home/name/logs/Aug.log
to avoid picking up the symlink.

This pull request shall resolve the issue once released: https://github.com/elastic/beats/pull/1767

Thanks @ruflin