Filebeat repeatedly sending old entries in log file

Elastic Stack Beats
1

Hello - just started using filebeat in a very simple setup and am seeing some odd behavior. My configuration is as follows:

filebeat:
  prospectors:
    -
      paths:
        - /usr/src/app/*.log
      input_type: log
      exclude_files: [".gz$"]
  registry_file: "/usr/src/app/filebeat/registry"

output:
  logstash:
    hosts: ["myserver.com:10200"]
    index: log

shipper:

logging:
  files:
    path: /var/log/filebeat/

The behavior I'm noticing is that every time I manually modify a log file every line in the file is sent to logstash. Meaning if I have a log file with two lines in it that have already been logged, then I add one more entry and save, all three lines are sent. It looks like the registry is properly updated - the offset changes each time I modify the file.

Probably something stupid I'm doing - any suggestions?

Here's the registry file:

{"/usr/src/app/log.log":{"source":"/usr/src/app/log.log","offset":400,"FileStateOS":{"inode":663,"device":31}},"/usr/src/app/log2.log":{"source":"/usr/src/app/log2.log","offset":180,"FileStateOS":{"inode":618,"device":31}}}
1 Like
2

how exactly do you modify the file. Have you tried $ echo 'new line' >> /usr/src/app/log.log? filebeat tries to detect if file is new or not based on meta-data. some tools/editors do create a completely new file on save and unlink the old one. This is detected by filebeat as new file forcing it to resend all content.

2 Likes
3

I was using vi to edit the file manually - that's probably the case then. It ended up not being a problem when I started appending to the log files using node.

1 Like
4

This topic was automatically closed after 21 days. New replies are no longer allowed.