Filebeat resends log files after restart

Dear colleagues,
I have an issue with Filebeat again - I've noticed that it resends the whole log files after restarting the application despite the log file stating that the registry file has been loaded successfully.
Here's a snippet of my config:

- type: filestream
  enabled: true
   - /path/to/logs/*.log
  - multiline:
      type: pattern
      pattern: '^[0-9]{4}[[:space:]]+[A-Z]{1}[a-z]{2}[[:space:]]+[0-9]{2}'
      negate: True
      match: after
  name: tibco_int
  tags: ["tibcoint1-logs"]
  ignore_older: 24h

The command I use to run filebeat:
./filebeat -c /path_to_cfg/elk_cfg/filebeat_cfg/filebeat_ssl.yml --path.config /path_to_cfg/elk_cfg/filebeat_cfg/ /path_to_data/elk_data/filebeat

The steps to reproduce this:
1.Run Filebeat
2.Insert some example log line to an existing log file from the config directory.
3.Restart FIlebeat
4.Result: all of the files with the modification date <24h are resent to the configured output.

The setting: ignore_older: 5s works and resolves the issue, but I want to know why Filebeat resends the whole log files instead of picking up on the last line? Isn't that the point of having the registry configured?
Below a snippet of a log file after point 3.:

First line of the log:

{"log.level":"info","@timestamp":"2022-05-04T11:40:49.839+0200","log.origin":{"":"instance/beat.go","file.line":669},"message":"Home path: [/srv/int/filebeat-8.1.1-linux-x86_64] Config path: [/srv/int/elk_cfg/filebeat_cfg/] Data path: [/srv/int/elk_data/filebeat] Logs path: [/srv/int/filebeat-8.1.1-linux-x86_64/logs]","":"filebeat","ecs.version":"1.6.0"}

Some lines after we have this:

{"log.level":"info","@timestamp":"2022-05-04T11:40:49.894+0200","log.origin":{"":"memlog/store.go","file.line":124},"message":"Finished loading transaction log file for '/srv/int/elk_data/filebeat/registry/filebeat'. Active transaction id=6161","":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-04T11:40:49.894+0200","log.origin":{"":"beater/filebeat.go","file.line":285},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-04T11:40:49.895+0200","log.logger":"registrar","log.origin":{"":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","":"filebeat","ecs.version":"1.6.0"}

I have another case of this:
Filebeat resends the whole log file, without restarting, if the 24h period is exceeded (no logs during last 24h), then if a new log entry is saved to the file, Filebeat resends the whole log file.
Note that the logs are inserted by an application automatically, not manually by me.

how to solve this ?

I'm seeing this behavior too, be nice if someone from Elastic commented, this seems to be contrary to what filebeat is supposed to be doing.

From what I understand, there must be some kind of mechanism that causes Filebeat to clear the registry for files after some period of time.
Indeed it would be nice if someone from Elastic explained as this is not clear from just the documentation. My colleague from the same company also has this issue in a completely different environment, so this problem definitely exists.

What version of Filebeat are you running?

Could you please open an issue on GH? It seems to be a bug.

I'm currently running 8.1.1, but the problem existed on previous <8 versions.
I've just opened this issue:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.