Filebeat not harvesting newly added records

Hi,I have a filebeat which is running on windows server 2019.The data is actively getting written to that log file but it's timestamp changes every 30 mins.I trying to reading log file,but for some reason the newly added records are not getting ingested?

this is the filebeat config:

type: filestream
enabled: true
paths : 'C:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log'
tags: ['console_data']

filebeat debug logs

2023-04-24T06:19:33.495Z INFO [file_watcher] filestream/fswatch.go:137 Start next scan
2023-04-24T06:19:33.496Z DEBUG [file_watcher] filestream/fswatch.go:204 Found 1 paths
2023-04-24T06:19:33.496Z DEBUG [input.filestream] filestream/prospector.go:164 File C:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log has been updated {"id": "41C27034C04F35E3", "prospector": "file_prospector", "operation": "write", "source_name": "native::153485312-87546-3234102977", "os_id": "153485312-87546-3234102977", "new_path": "C:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log", "old_path": "C:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log"}
2023-04-24T06:19:33.496Z DEBUG [input.filestream] input-logfile/harvester.go:145 Starting harvester for file {"id": "41C27034C04F35E3", "source": "filestream::.global::native::153485312-87546-3234102977"}
2023-04-24T06:19:33.497Z DEBUG [input.filestream] input-logfile/harvester.go:181 Stopped harvester for file {"id": "41C27034C04F35E3", "source": "filestream::.global::native::153485312-87546-3234102977"}
2023-04-24T06:19:43.322Z DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: C:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log; Backoff now. {"id": "41C27034C04F35E3", "source": "filestream::.global::native::153485312-87546-3234102977", "path": "C:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log", "state-id": "native::153485312-87546-3234102977"}

Hi @Dasher

Are they getting ingested after the 30 minutes passes and they are no longer new? Or in other words does the new data eventually make it to the cluster, but is effectively on a 30 minute delay?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.