Filebeat sending whole log everytime instead of just newly added lines

[elk11]

Hi,

Filebeat is sending whole log every time for addition of even one new line in log file. because of this I'm ending up with lot of duplicate data.

Below is my config file:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/logs/zuul/log/50lk_1cr_lines.log
  encoding: plain
  include_lines: ['\d{4}\-\d{2}\-\d{2}']
  multiline.pattern: '^\d{4}\-\d{2}\-\d{2}'
  multiline.negate: false
  multiline.match: after
  fields:
    document_type: zuul_log
  clean_removed: true
  registry: /usr/share/filebeat/data/registry

setup.template:
  name: "filebeat"
  pattern: "filebeat-*"
  settings:
    index.number_of_shards: 1
    index.number_of_replicas: 1

output.elasticsearch:
  hosts: ["xxx:9200"]
  index: "%{[fields.document_type]}-%{[beat.version]}-%{+yyyy.MM.dd}"

[elk11]

how exactly do you modify the file. Have you tried $ echo 'new line' >> /usr/src/app/log.log ? filebeat tries to detect if file is new or not based on meta-data. some tools/editors do create a completely new file on save and unlink the old one. This is detected by filebeat as new file forcing it to resend all content.

I was using vi editor to edit file manually. so the issue. The above solved problem.