Dear colleagues,
I have an issue with Filebeat again - I've noticed that it resends the whole log files after restarting the application despite the log file stating that the registry file has been loaded successfully.
Here's a snippet of my config:
filebeat.inputs:
- type: filestream
enabled: true
paths:
- /path/to/logs/*.log
parsers:
- multiline:
type: pattern
pattern: '^[0-9]{4}[[:space:]]+[A-Z]{1}[a-z]{2}[[:space:]]+[0-9]{2}'
negate: True
match: after
name: tibco_int
tags: ["tibcoint1-logs"]
ignore_older: 24h
The command I use to run filebeat:
./filebeat -c /path_to_cfg/elk_cfg/filebeat_cfg/filebeat_ssl.yml --path.config /path_to_cfg/elk_cfg/filebeat_cfg/ --path.data /path_to_data/elk_data/filebeat
The steps to reproduce this:
1.Run Filebeat
2.Insert some example log line to an existing log file from the config directory.
3.Restart FIlebeat
4.Result: all of the files with the modification date <24h are resent to the configured output.
The setting: ignore_older: 5s works and resolves the issue, but I want to know why Filebeat resends the whole log files instead of picking up on the last line? Isn't that the point of having the registry configured?
Below a snippet of a log file after point 3.:
First line of the log:
{"log.level":"info","@timestamp":"2022-05-04T11:40:49.839+0200","log.origin":{"file.name":"instance/beat.go","file.line":669},"message":"Home path: [/srv/int/filebeat-8.1.1-linux-x86_64] Config path: [/srv/int/elk_cfg/filebeat_cfg/] Data path: [/srv/int/elk_data/filebeat] Logs path: [/srv/int/filebeat-8.1.1-linux-x86_64/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
Some lines after we have this:
{"log.level":"info","@timestamp":"2022-05-04T11:40:49.894+0200","log.origin":{"file.name":"memlog/store.go","file.line":124},"message":"Finished loading transaction log file for '/srv/int/elk_data/filebeat/registry/filebeat'. Active transaction id=6161","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-04T11:40:49.894+0200","log.origin":{"file.name":"beater/filebeat.go","file.line":285},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-04T11:40:49.895+0200","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}