The problem that we are facing is, multiple harvester is being created for a single log file, within a span of 7 seconds as result of which duplicate logs being generated from filebeat.
For example, please find the below set of logs flow from log producer to filebeat
Log Producer microservice logs -test-controller
Single logs being printed in log generator microservice container logs with message = "auto,1.64"
{"timestamp":"2024-05-10T05:54:51.350+02:00","service_id":"test-controller","message":"auto,1.64","metadata":{"category":"HA-in-service-performance","namespace":"test","pod_name":"test-controller-6f565dd5b9-qv2w2","application_id":"test884"},"severity":"info","version":"1.1.0","facility":"security/authorization messages","extra_data":{"test_stream":"dsp","in_service_performance":{"version": "1.0.0","originating_service_name": "test-controller","originating_service_version": "1.239.7-1","originating_pod_name": "test-controller-6f565dd5b9-qv2w2","event_type":"small-local-restart","reporting_service_version": "1.239.7-1"}}}
But in the configured output we are seeing the logs being seen twice, sometime 4 times. while we are analysing the beats logs we could see two harvesters being created. please find the filebeat logs as mentioned below for the same file path being started twice(test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log)
{"log.level":"info","@timestamp":"2024-05-10T04:56:47.874+0200","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":310},"message":"Harvester started for paths: [/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log]","service.name":"filebeat","input_id":"e271da1d-390e-48ac-be82-4be7d7e644bc","source_file":"/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log","state_id":"native::143563266-64515","finished":false,"os_id":"14266-64515","harvester_id":"e9f36c6a-d5c1-4ddc-958f-d254da5b6ea6","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-05-10T04:56:54.160+0200","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":310},"message":"Harvester started for paths: [/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log]","service.name":"filebeat","input_id":"71aa7599-3723-4e6c-a3d2-f9ac0a1007ec","source_file":"/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log","state_id":"native::14266-64515","finished":false,"os_id":"14266-64515","harvester_id":"2e86644e-e9c7-4f68-b859-b225711f9220","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-05-10T04:56:54.281+0200","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":337},"message":"Reader was closed. Closing.","service.name":"filebeat","input_id":"71aa7599-3723-4e6c-a3d2-f9ac0a1007ec","source_file":"/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log","state_id":"native::14266-64515","finished":false,"os_id":"143563266-64515","harvester_id":"2e86644e-e9c7-4f68-b859-b225711f9220","ecs.version":"1.6.0"}
Hence we wanted to understand this is causing duplicate logs being sent to the configured output? Why multiple harvester is being created? is there any way we can avoid it?
NOTE: there are no restarts or disturbances caused to the registry file