# FileBeat loses 10 seconds of logs on rotation
**URL:** https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227
**Category:** Beats
**Tags:** filebeat
**Created:** [August 23, 2021, 1:04pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227 "2021-08-23T13:04:17Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@jj12341](https://discuss.elastic.co/u/jj12341)
#### Post date: [August 23, 2021, 1:04pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/1 "2021-08-23T13:04:17Z")
Hi!
My FileBeat is losing 10 seconds of approximately logs. Obviously the problem is in the rotation of the files, but I can not find the correct option so that these logs are not lost.
The Rotation of Logs performs Log4J:
```auto
655947 -rw-r--r-- 1 user group 21M Aug 20 09:34 /opt/weblogic/logs/app/app-150.log
655948 -rw-r--r-- 1 user group 20M Aug 20 18:19 /opt/weblogic/logs/app/app.log
----- rotated -----
655948 -rw-r--r-- 1 user group 21M Aug 20 09:34 /opt/weblogic/logs/app/app-150.log
655949 -rw-r--r-- 1 user group 20M Aug 20 18:19 /opt/weblogic/logs/app/app.log
```
By listing the Inodes, when rotating the log, the inode is maintained for the rotated log (150) so I understand that FileBeat should read it until the end but it does not happen.
In the log of FileBeat it is indicated that it was truncated and starts again to read the file without finishing reading the 150. Is this my basic configuration, any idea?
```auto
- type: log
enabled: true
fields:
log: "app"
server: "server4"
paths:
- /opt/weblogic/logs/app/app.log
```
Should I use any of these options: scan\_frecuency, close\_inactive, harvester\_buffer\_size ??
or maybe use the path of the log and the rotated, like:
```auto
paths:
- /opt/weblogic/logs/app/app-150.log
- /opt/weblogic/logs/app/app.log
```
Thanks for the help!
---
### Author:  [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [August 24, 2021, 10:53am UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/2 "2021-08-24T10:53:14Z")
I would use a path like `/opt/weblogic/logs/app/app*.log` or else it no longer matches and filebeat may stop caring about it even though it wasn't done reading it.
---
### Author:  [@jj12341](https://discuss.elastic.co/u/jj12341)
#### Post date: [August 24, 2021, 12:13pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/3 "2021-08-24T12:13:52Z")
> [@legoguy1000](#):
>
> I would use a path like `/opt/weblogic/logs/app/app*.log` or else it no longer matches and filebeat may stop caring about it even though it wasn't done reading it.
Thinking the same I did the test of using the paths:
```auto
/opt/weblogic/logs/app/app.log
/opt/weblogic/logs/app/app-150.log (rotated)
```
But it had duplicates every time it rotated. Is there a difference if I use the wildcard? (there are 150 files per server)
---
### Author:  [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [August 24, 2021, 12:30pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/4 "2021-08-24T12:30:55Z")
Idk if it's treated differently. It may be since it's the same glob as opposed to 2 different paths, but that's just a guess. I'd try just to see.
---
### Author:  [@jj12341](https://discuss.elastic.co/u/jj12341)
#### Post date: [August 24, 2021, 1:42pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/5 "2021-08-24T13:42:37Z")
```auto
- type: log
enabled: true
close_inactive: 5m
fields:
log: "app"
server: "server4"
paths:
- /opt/weblogic/logs/app/app*.log
```
I tried this configuration but I keep losing the end of the rotated files and I don't see it closing the inactive ones for more than 5 minutes. Also I have the following error in the log:
`2021-08-24T13:39:12.465Z ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: 429 Too Many Requests: 429 Too Many Requests /_bulk `
---
### Author:  [@jj12341](https://discuss.elastic.co/u/jj12341)
#### Post date: [August 25, 2021, 12:13pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/6 "2021-08-25T12:13:16Z")
I changed my settings to the following ... which I think is the right thing to do, but I keep wasting those seconds at the end of each rotated log:
```auto
- type: log
enabled: true
harvester_limit: 10
ignore_older: 72h
close_inactive: 5m
clean_inactive: 74h
fields:
log: "app"
server: "server4"
paths:
- /opt/weblogic/logs/app/app*.log
```
(ps: I am no longer having truncated messages since I remove close\_rename)
ps2: The log usually rotates frequently ... but it never takes the last lines of the rotated log ... even though the harvest is alive.
---
### Author:  [@jj12341](https://discuss.elastic.co/u/jj12341)
#### Post date: [August 25, 2021, 1:52pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/8 "2021-08-25T13:52:14Z")
this happens frequently ... is it possible that it is closing before finishing reading the log? :
```auto
2021-08-25T13:42:04.536Z INFO log/harvester.go:333 File is inactive: /logs/app-150.log. Closing because close_inactive of 20m0s reached.
2021-08-25T13:42:04.536Z INFO log/harvester.go:333 File is inactive: /logs/app-149.log. Closing because close_inactive of 20m0s reached.
2021-08-25T13:42:04.569Z INFO log/harvester.go:333 File is inactive: /logs/app-145.log. Closing because close_inactive of 20m0s reached.
2021-08-25T13:42:04.569Z INFO log/harvester.go:333 File is inactive: /logs/app-146.log. Closing because close_inactive of 20m0s reached.
2021-08-25T13:42:11.174Z INFO log/harvester.go:302 Harvester started for file: /logs/app-145.log
2021-08-25T13:42:11.175Z INFO log/harvester.go:302 Harvester started for file: /logs/app-146.log
2021-08-25T13:42:11.176Z INFO log/harvester.go:302 Harvester started for file: /logs/app-149.log
2021-08-25T13:42:11.176Z INFO log/harvester.go:302 Harvester started for file: /logs/app-150.log
```
---
### Author:  [@jj12341](https://discuss.elastic.co/u/jj12341)
#### Post date: [August 26, 2021, 12:19pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/9 "2021-08-26T12:19:15Z")
up ?
---
### Author:  [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [August 27, 2021, 11:06am UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/10 "2021-08-27T11:06:31Z")
One thing I saw in another post similar to this is does the final log entry end with a new line character, `\n`? Without it filebeat waits expecting more to be written.
---
### Author:  [@jj12341](https://discuss.elastic.co/u/jj12341)
#### Post date: [August 27, 2021, 1:56pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/11 "2021-08-27T13:56:52Z")
Apparently my rotation of logs was making disasters with the inodes ... if someone reaches this post I suggest you check that. With an `ls -li` to see how the inodes and file names in the logs change.
Greetings!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 24, 2021, 3:57pm UTC](https://discuss.elastic.co/t/filebeat-loses-10-seconds-of-logs-on-rotation/282227/12 "2021-09-24T15:57:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.