hi,
We found some duplicate events in Elasticsearch and those events were from the renamed log file.
Filebeat version : 7.17
OS: Windows 2019 server
Here is the filebeat log:
2024-05-28T09:02:42.320+0800 INFO [input.filestream] filestream/input.go:321 Reader was closed. Closing. {"id": "AED3DB69868B5069", "source": "filestream::.global::native::8716288-126642-177104865", "path": "D:\\FAB300\\Log\\csimweb_20240525.21.18.18.11_kcmes1_32168.log", "state-id": "native::8716288-126642-177104865"}
2024-05-28T09:11:03.658+0800 INFO [input.filestream] filestream/input.go:321 Reader was closed. Closing. {"id": "AED3DB69868B5069", "source": "filestream::.global::native::8716288-126642-177104865", "path": "D:\\FAB300\\Log\\csimweb_20240525.21.18.18.11_kcmes1_32168.log", "state-id": "native::8716288-126642-177104865"}
2024-05-28T09:14:28.233+0800 INFO [input.filestream] filestream/input.go:321 Reader was closed. Closing. {"id": "AED3DB69868B5069", "source": "filestream::.global::native::8716288-126642-177104865", "path": "D:\\FAB300\\Log\\csimweb_20240525.21.18.18.11_kcmes1_32168_20240528.09.11.03.63.log", "state-id": "native::8716288-126642-177104865"}
The file id "native::8716288-126642-177104865" was the same after the file rename.
Why Filebeat harvest the renamed log file again?
Is the only way to prevent this situation is using fingerprint?