# Filebeat resends log files after restart
**URL:** https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924
**Category:** Beats
**Tags:** filebeat
**Created:** [May 4, 2022, 9:43am UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924 "2022-05-04T09:43:19Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@alytkowski](https://discuss.elastic.co/u/alytkowski)
#### Post date: [May 4, 2022, 9:43am UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/1 "2022-05-04T09:43:19Z")
Dear colleagues,
I have an issue with Filebeat again - I've noticed that it resends the whole log files after restarting the application despite the log file stating that the registry file has been loaded successfully.
Here's a snippet of my config:
```auto
filebeat.inputs:
- type: filestream
enabled: true
paths:
- /path/to/logs/*.log
parsers:
- multiline:
type: pattern
pattern: '^[0-9]{4}[[:space:]]+[A-Z]{1}[a-z]{2}[[:space:]]+[0-9]{2}'
negate: True
match: after
name: tibco_int
tags: ["tibcoint1-logs"]
ignore_older: 24h
```
The command I use to run filebeat:
./filebeat -c /path\_to\_cfg/elk\_cfg/filebeat\_cfg/filebeat\_ssl.yml --path.config /path\_to\_cfg/elk\_cfg/filebeat\_cfg/ --path.data /path\_to\_data/elk\_data/filebeat
The steps to reproduce this:
1.Run Filebeat
2.Insert some example log line to an existing log file from the config directory.
3.Restart FIlebeat
4.Result: all of the files with the modification date \<24h are resent to the configured output.
The setting: ignore\_older: 5s works and resolves the issue, but I want to know why Filebeat resends the whole log files instead of picking up on the last line? Isn't that the point of having the registry configured?
Below a snippet of a log file after point 3.:
First line of the log:
```auto
{"log.level":"info","@timestamp":"2022-05-04T11:40:49.839+0200","log.origin":{"file.name":"instance/beat.go","file.line":669},"message":"Home path: [/srv/int/filebeat-8.1.1-linux-x86_64] Config path: [/srv/int/elk_cfg/filebeat_cfg/] Data path: [/srv/int/elk_data/filebeat] Logs path: [/srv/int/filebeat-8.1.1-linux-x86_64/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
```
Some lines after we have this:
```auto
{"log.level":"info","@timestamp":"2022-05-04T11:40:49.894+0200","log.origin":{"file.name":"memlog/store.go","file.line":124},"message":"Finished loading transaction log file for '/srv/int/elk_data/filebeat/registry/filebeat'. Active transaction id=6161","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-04T11:40:49.894+0200","log.origin":{"file.name":"beater/filebeat.go","file.line":285},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-04T11:40:49.895+0200","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
```
---
### Author:  [@alytkowski](https://discuss.elastic.co/u/alytkowski)
#### Post date: [May 5, 2022, 6:40am UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/2 "2022-05-05T06:40:32Z")
I have another case of this:
Filebeat resends the whole log file, without restarting, if the 24h period is exceeded (no logs during last 24h), then if a new log entry is saved to the file, Filebeat resends the whole log file.
Note that the logs are inserted by an application automatically, not manually by me.
---
### Author:  [@ren\_Da](https://discuss.elastic.co/u/ren_Da)
#### Post date: [May 25, 2022, 2:10am UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/3 "2022-05-25T02:10:54Z")
how to solve this ?
---
### Author:  [@jhbigler](https://discuss.elastic.co/u/jhbigler)
#### Post date: [June 9, 2022, 10:04pm UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/4 "2022-06-09T22:04:29Z")
I'm seeing this behavior too, be nice if someone from Elastic commented, this seems to be contrary to what filebeat is supposed to be doing.
---
### Author:  [@alytkowski](https://discuss.elastic.co/u/alytkowski)
#### Post date: [June 10, 2022, 5:52am UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/5 "2022-06-10T05:52:24Z")
From what I understand, there must be some kind of mechanism that causes Filebeat to clear the registry for files after some period of time.
Indeed it would be nice if someone from Elastic explained as this is not clear from just the documentation. My colleague from the same company also has this issue in a completely different environment, so this problem definitely exists.
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [June 13, 2022, 1:52pm UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/6 "2022-06-13T13:52:36Z")
What version of Filebeat are you running?
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [June 13, 2022, 2:07pm UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/7 "2022-06-13T14:07:24Z")
Could you please open an issue on GH? It seems to be a bug.
---
### Author:  [@alytkowski](https://discuss.elastic.co/u/alytkowski)
#### Post date: [June 14, 2022, 6:04am UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/8 "2022-06-14T06:04:09Z")
I'm currently running 8.1.1, but the problem existed on previous \<8 versions.
I've just opened this issue:
>
>
> First case:
> Filebeat resends the whole log files after restarting despite the l…og file stating that the registry file has been loaded successfully.
> Second case:
> Filebeat resends the whole log file, without restarting Filebeat, if the 24h period is exceeded (no logs during last 24h), then if a new log entry is saved to the file, Filebeat resends the whole log file.
>
> For confirmed bugs, please report:
> \- Version: 8.1.1
> \- Operating System: RHEL 7.9
> \- Discuss Forum URL: https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/5
> \- Steps to Reproduce:
> 1.Run Filebeat
> 2.Insert some example log line to an existing log file from the config directory.
> 3.Restart FIlebeat
> 4.Result: all of the files with the modification date \<24h are resent to the configured output.
>
> 1.Run Filebeat
> 2.Insert some example log line to an existing log file from the config directory.
> 3.Wait for the period set in "ignore\_older" to pass and insert a new line to the log file after.
> 4.Result: all of the files with the modification date \<24h are resent to the configured output.
>
> \-Snippet of configuration
> \`filebeat.inputs:
> \- type: filestream
> enabled: true
> paths:
> - /path/to/logs/\*.log
> parsers:
> - multiline:
> type: pattern
> pattern: '^\[0-9\]{4}\[\[:space:\]\]+\[A-Z\]{1}\[a-z\]{2}\[\[:space:\]\]+\[0-9\]{2}'
> negate: True
> match: after
> name: tibco\_int
> tags: \["tibcoint1-logs"\]
> ignore\_older: 24h\`
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 12, 2022, 8:04am UTC](https://discuss.elastic.co/t/filebeat-resends-log-files-after-restart/303924/9 "2022-07-12T08:04:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.