I'm using Filebeat 7.16.2 to ship logs to Logstash -> ES.
For some reason every time I restart filebeat service, it scans all files in the log directory again and sends them to logstash. registry directory is accessible and writable and I can see it being updated.
Here is the config files:
/etc/filebeat/filebeat.yml
---
name: testapp.company.com
tags: []
fields: {}
fields_under_root: false
filebeat:
config.inputs:
enabled: true
path: "/etc/filebeat/conf.d/*.yml"
config.modules:
enabled: false
path: "/etc/filebeat/modules.d/*.yml"
shutdown_timeout: '10s'
modules: []
registry:
path: "/var/lib/filebeat"
file_permissions: '0600'
flush: 0s
http: {}
cloud: {}
output:
logstash:
hosts:
- logstash.example.com:9008
shipper: {}
logging:
level: info
to_files: true
files:
path: "/var/log/filebeat"
name: filebeat
keepfiles: 7
permissions: 416
autodiscover: {}
runoptions: {}
processors: []
setup: {}
/etc/filebeat/conf.d/app.yml
---
- type: filestream
paths:
- /var/log/app/2022/*/*.log
encoding: plain
exclude_lines:
- 'DEBUG'
- 'INFO'
- 'NOTICE'
parsers:
- multiline:
type: pattern
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
max_lines: 150
timeout: '10s'
fields:
type: app
env: dev
Registry dir:
ls -la /var/lib/filebeat/
total 16
drwxrwxrwx 3 root root 4096 Jan 10 16:42 .
drwxr-xr-x 51 root root 4096 Jul 25 2019 ..
drwxr-x--- 2 root root 4096 Jan 10 15:27 filebeat
-rw------- 1 root root 0 Jan 10 16:42 filebeat.lock
-rw------- 1 root root 95 Jan 10 15:27 meta.json
ls -la /var/lib/filebeat/filebeat
total 148
drwxr-x--- 2 root root 4096 Jan 10 15:27 .
drwxrwxrwx 3 root root 4096 Jan 10 16:42 ..
-rw------- 1 root root 131683 Jan 10 16:47 log.json
-rw------- 1 root root 15 Jan 10 15:27 meta.json