Is it expected that filestream input picks up all documents again when file is updated again after close.on_state_change.inactive is exceeded?

Elastic Stack Beats
1

Hey all,

I'm using Filebeat 8.11.3 and I noticed that it sometimes resubmits documents that it already handled before.
So I'm wondering what the expected behaviour is of Filebeat.
There is a setting filestream input | Filebeat Reference [8.11] | Elastic.

How I think it works in this version is that the filestream input starts reading a file. In case it doesn't receive any new documents anymore in that file and the close.on_state_change.inactive time is exceeded, filebeat closes the handles for that file.
That makes sense to me.
But when there are new documents again after this time is exceeded, filebeat reprocesses all documents again.

Is this expected behavior? If not, how is it supposed to work?

Just for reference, this is input config I'm using:

filebeat:
    inputs:
    -   close.on_state_change.inactive: 10m
        enabled: true
        id: some_id_log
        paths:
        - /data/log/app/appparsed/*/app-*.log*
        prospector.scanner.exclude_files:
        - \.gz$
        type: filestream

10 minutes before the first log below, the logs were stopped and the handles was closed as expected.
Filebeat logs:

{"log.level":"info","@timestamp":"2024-03-14T08:52:55.387+0100","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/data/log/app/appparsed/TEST-8-2-B-FGPVRPP-P-NI/app-TEST-8-2-B-FGPVRPP-P-NI.log'","service.name":"filebeat","id":"some_id_log","source_file":"filestream::some_id_log::native::536879026-64770","path":"/data/log/app/appparsed/TEST-8-2-B-FGPVRPP-P-NI/app-TEST-8-2-B-FGPVRPP-P-NI.log","state-id":"native::536879026-64770","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-14T08:53:20.378+0100","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/data/log/app/appparsed/TEST-8-2-B-FGPVRPP-P-NI/app-TEST-8-2-B-FGPVRPP-P-NI.log","service.name":"filebeat","id":"some_id_log","source_file":"filestream::some_id_log::native::536879026-64770","path":"/data/log/app/appparsed/TEST-8-2-B-FGPVRPP-P-NI/app-TEST-8-2-B-FGPVRPP-P-NI.log","state-id":"native::536879026-64770","ecs.version":"1.6.0"}

Thanks in advance!

2

Is this a network share or a path in the disk?

4

It is a local path, XFS filesystem

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.