Filebeat repeatedly sending old entries in log file

(James Trachy) #1

Hello - just started using filebeat in a very simple setup and am seeing some odd behavior. My configuration is as follows:

        - /usr/src/app/*.log
      input_type: log
      exclude_files: [".gz$"]
  registry_file: "/usr/src/app/filebeat/registry"

    hosts: [""]
    index: log


    path: /var/log/filebeat/

The behavior I'm noticing is that every time I manually modify a log file every line in the file is sent to logstash. Meaning if I have a log file with two lines in it that have already been logged, then I add one more entry and save, all three lines are sent. It looks like the registry is properly updated - the offset changes each time I modify the file.

Probably something stupid I'm doing - any suggestions?

Here's the registry file:


(Steffen Siering) #2

how exactly do you modify the file. Have you tried $ echo 'new line' >> /usr/src/app/log.log? filebeat tries to detect if file is new or not based on meta-data. some tools/editors do create a completely new file on save and unlink the old one. This is detected by filebeat as new file forcing it to resend all content.

(James Trachy) #3

I was using vi to edit the file manually - that's probably the case then. It ended up not being a problem when I started appending to the log files using node.

(system) #4

This topic was automatically closed after 21 days. New replies are no longer allowed.