How to send only new line data in filebeats

hi i try to config filebeat like a if in CSV or log file it have new line record it must to send only new line record not all off them data
this is my config

- type: log
  enabled: true
      - D:\Filebeat\datafrombeat.csv
    #- c:\programdata\elasticsearch\logs\*
   ignore_older: 12h
   close_older: 1m
   close_inactive: 1m
   clean_removed: true
   close_removed: true
   scan_frequency: 10s
   registry_file: registry

thank you

Are you saying that Filebeat is resending the whole file rather than just the newly added log lines?

If so, that would mean that Filebeat is not "remembering" that it's reading from the same file. So either it's not the same file (the whole file is being re-written each time it's updated and the file has a new inode) OR Filebeat is cleaning its persisted state for the inode at some point.

How is the CSV file being updated? Filebeat expects log files to be appended to.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.