I'm having problem with filebeat - filebeat send old logs after service restart. I've turned on debugging and got:
2016-07-01T09:42:45+02:00 INFO Additional configs loaded from: /etc/filebeat/conf.d/maillog.yml
2016-07-01T09:42:45+02:00 DBG File Configs: [/var/log/maillog]
2016-07-01T09:42:45+02:00 DBG scan path /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Check file for harvesting: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Start harvesting unknown file: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Resuming harvester on a previously harvested file: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Registrar will re-save state for /var/log/maillog
2016-07-01T09:42:45+02:00 DBG scan path /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Check file for harvesting: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Update existing file for harvesting: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Not harvesting, file didn't change: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG harvest: "/var/log/maillog" position:19453749 (offset snapshot:0)
2016-07-01T09:42:45+02:00 INFO Harvester started for file: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG End of file reached: /var/log/maillog; Backoff now.
next many:
2016-07-01T09:42:45+02:00 DBG full line read
and then I see old logs published (with current logs also). In registry file offset was set to non-zero value for this file. I have something wrong in configuration or is it bug in filebeat?
My config:
/etc/filebeat/filebeat.yml
filebeat:
spool_size: 2048
publish_async: false
idle_timeout: 5s
registry_file: /var/lib/filebeat/registry
config_dir: /etc/filebeat/conf.d
output:
logstash:
hosts:
- c.y:5044
- d.y:5044
loadbalance: true
index: logstash
shipper:
name: a.b
logging:
to_syslog: false
to_files: true
files: path: /var/log/filebeat name: filebeat.log keepfiles: 7
level: warning
/etc/filebeat/conf.d/maillog.yml
filebeat:
prospectors:
- paths:
- /var/log/maillog
encoding: plain
fields_under_root: false
input_type: log
ignore_older: 5m
close_older: 5m
document_type: maillog
scan_frequency: 1s
harvester_buffer_size: 16384
tail_files: false
force_close_files: false
backoff: 1s
max_backoff: 10s
backoff_factor: 2
partial_line_waiting: 5s
max_bytes: 8192