Hey all,
I'm using Filebeat 8.11.3 and I noticed that it sometimes resubmits documents that it already handled before.
So I'm wondering what the expected behaviour is of Filebeat.
There is a setting filestream input | Filebeat Reference [8.11] | Elastic.
How I think it works in this version is that the filestream input starts reading a file. In case it doesn't receive any new documents anymore in that file and the close.on_state_change.inactive time is exceeded, filebeat closes the handles for that file.
That makes sense to me.
But when there are new documents again after this time is exceeded, filebeat reprocesses all documents again.
Is this expected behavior? If not, how is it supposed to work?
Just for reference, this is input config I'm using:
filebeat:
inputs:
- close.on_state_change.inactive: 10m
enabled: true
id: some_id_log
paths:
- /data/log/app/appparsed/*/app-*.log*
prospector.scanner.exclude_files:
- \.gz$
type: filestream
10 minutes before the first log below, the logs were stopped and the handles was closed as expected.
Filebeat logs:
{"log.level":"info","@timestamp":"2024-03-14T08:52:55.387+0100","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/data/log/app/appparsed/TEST-8-2-B-FGPVRPP-P-NI/app-TEST-8-2-B-FGPVRPP-P-NI.log'","service.name":"filebeat","id":"some_id_log","source_file":"filestream::some_id_log::native::536879026-64770","path":"/data/log/app/appparsed/TEST-8-2-B-FGPVRPP-P-NI/app-TEST-8-2-B-FGPVRPP-P-NI.log","state-id":"native::536879026-64770","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-14T08:53:20.378+0100","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/data/log/app/appparsed/TEST-8-2-B-FGPVRPP-P-NI/app-TEST-8-2-B-FGPVRPP-P-NI.log","service.name":"filebeat","id":"some_id_log","source_file":"filestream::some_id_log::native::536879026-64770","path":"/data/log/app/appparsed/TEST-8-2-B-FGPVRPP-P-NI/app-TEST-8-2-B-FGPVRPP-P-NI.log","state-id":"native::536879026-64770","ecs.version":"1.6.0"}
Thanks in advance!