Filebeat sends old logs after reboot


(Lucas) #1

I'm having problem with filebeat - filebeat send old logs after service restart. I've turned on debugging and got:

2016-07-01T09:42:45+02:00 INFO Additional configs loaded from: /etc/filebeat/conf.d/maillog.yml
2016-07-01T09:42:45+02:00 DBG File Configs: [/var/log/maillog]
2016-07-01T09:42:45+02:00 DBG scan path /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Check file for harvesting: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Start harvesting unknown file: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Resuming harvester on a previously harvested file: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Registrar will re-save state for /var/log/maillog
2016-07-01T09:42:45+02:00 DBG scan path /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Check file for harvesting: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Update existing file for harvesting: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG Not harvesting, file didn't change: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG harvest: "/var/log/maillog" position:19453749 (offset snapshot:0)
2016-07-01T09:42:45+02:00 INFO Harvester started for file: /var/log/maillog
2016-07-01T09:42:45+02:00 DBG End of file reached: /var/log/maillog; Backoff now.

next many:

2016-07-01T09:42:45+02:00 DBG full line read

and then I see old logs published (with current logs also). In registry file offset was set to non-zero value for this file. I have something wrong in configuration or is it bug in filebeat?

My config:

/etc/filebeat/filebeat.yml
filebeat:
spool_size: 2048
publish_async: false
idle_timeout: 5s
registry_file: /var/lib/filebeat/registry
config_dir: /etc/filebeat/conf.d

output:
logstash:
hosts:
- c.y:5044
- d.y:5044
loadbalance: true
index: logstash

shipper:
name: a.b

logging:
to_syslog: false
to_files: true

files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 7
level: warning

/etc/filebeat/conf.d/maillog.yml
filebeat:
prospectors:
- paths:
- /var/log/maillog
encoding: plain
fields_under_root: false
input_type: log
ignore_older: 5m
close_older: 5m
document_type: maillog
scan_frequency: 1s
harvester_buffer_size: 16384
tail_files: false
force_close_files: false
backoff: 1s
max_backoff: 10s
backoff_factor: 2
partial_line_waiting: 5s
max_bytes: 8192


Filebeat and busy files
(system) #2

This topic was automatically closed after 21 days. New replies are no longer allowed.