Filebeat duplicates events after restart

I'm using Filebeat 7.16.2 to ship logs to Logstash -> ES.

For some reason every time I restart filebeat service, it scans all files in the log directory again and sends them to logstash. registry directory is accessible and writable and I can see it being updated.

Here is the config files:

/etc/filebeat/filebeat.yml

---
name: testapp.company.com
tags: []
fields: {}
fields_under_root: false
filebeat:
  config.inputs:
    enabled: true
    path: "/etc/filebeat/conf.d/*.yml"
  config.modules:
    enabled: false
    path: "/etc/filebeat/modules.d/*.yml"
  shutdown_timeout: '10s'
  modules: []
  registry:
    path: "/var/lib/filebeat"
    file_permissions: '0600'
    flush: 0s
http: {}
cloud: {}
output:
  logstash:
    hosts:
    - logstash.example.com:9008
shipper: {}
logging:
  level: info
  to_files: true
  files:
    path: "/var/log/filebeat"
    name: filebeat
    keepfiles: 7
    permissions: 416
autodiscover: {}
runoptions: {}
processors: []
setup: {}

/etc/filebeat/conf.d/app.yml

---
- type: filestream
  paths:
    - /var/log/app/2022/*/*.log
  encoding: plain
  exclude_lines:
    - 'DEBUG'
    - 'INFO'
    - 'NOTICE'
  parsers:
    - multiline:
        type: pattern
        pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
        negate: true
        match: after
        max_lines: 150
        timeout: '10s'
  fields:
    type: app
    env: dev

Registry dir:

ls -la /var/lib/filebeat/
total 16
drwxrwxrwx  3 root root 4096 Jan 10 16:42 .
drwxr-xr-x 51 root root 4096 Jul 25  2019 ..
drwxr-x---  2 root root 4096 Jan 10 15:27 filebeat
-rw-------  1 root root    0 Jan 10 16:42 filebeat.lock
-rw-------  1 root root   95 Jan 10 15:27 meta.json

ls -la /var/lib/filebeat/filebeat
total 148
drwxr-x--- 2 root root   4096 Jan 10 15:27 .
drwxrwxrwx 3 root root   4096 Jan 10 16:42 ..
-rw------- 1 root root 131683 Jan 10 16:47 log.json
-rw------- 1 root root     15 Jan 10 15:27 meta.json

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.