Hi
We are using ELK 8.11.1 we are seeing these types of errors logs being generated from filebeat.
filestream input with ID <> already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input.
2024-02-14T21:40:35.840910409Z stderr F {"log.level":"error","@timestamp":"2024-02-14T21:40:35.840Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":183},"message":"filestream input with ID 'kubernetes-container-logs-mon-metricbeat-wv68v-978c73fbd06bfead737e1a56ffe8705accc188317ed9e15d0fa3fef275839e83' already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input.","service.name":"filebeat","ecs.version":"1.6.0"}
2024-02-14T21:40:35.84478493Z stderr F {"log.level":"error","@timestamp":"2024-02-14T21:40:35.844Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":183},"message":"filestream input with ID 'kubernetes-container-logs-mon-metricbeat-wv68v-aa21559a0a537be343a4488d48cfb508eb3201be22989e980a1f03f186d2e322' already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input.","service.name":"filebeat","ecs.version":"1.6.0"}
2024-02-14T21:40:35.848644233Z stderr F {"log.level":"error","@timestamp":"2024-02-14T21:40:35.848Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":183},"message":"filestream input with ID 'kubernetes-container-logs-mon-metricbeat-wv68v-788bfb798144f391e7ea1e702b691b71211a4fa1456e883b61031aa8e07b0225' already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input.","service.name":"filebeat","ecs.version":"1.6.0"}
This is our config
filebeat.autodiscover:
providers:
- add_resource_metadata:
cronjob: false
deployment: false
hints.default_config:
close.on_state_change.renamed: true
id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
parsers:
- container: null
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
prospector.scanner:
fingerprint.enabled: true
symlinks: true
type: filestream
hints.enabled: true
host: ${NODE_NAME}
type: kubernetes
filebeat.inputs:
- close.reader.after_interval: 5m
file_identity.fingerprint: null
id: wra-filestream-id
paths:
- /var/log/*.log
- /var/log/messages
- /var/log/syslog
- /var/log/**/*.log
prospector.scanner.exclude_files:
- ^/var/log/containers/
- ^/var/log/pods/
prospector.scanner.fingerprint:
enabled: true
length: 280
offset: 0
type: filestream
Our /var/log/containers files are symlinks of the pod's logs (/var/log/pods/*)
ls -rtl /var/log/containers
total 448
lrwxrwxrwx 1 root root 125 Feb 13 02:39 kube-controller-manager-controller-0_kube-system_kube-controller-manager-b2b7024cdcaed8277f81936c466043d797219ff1d65bed2822755287f7ffdbbc.log -> /var/log/pods/kube-system_kube-controller-manager-controller-0_79fcaab94d97fd191c9e53535451309b/kube-controller-manager/1.log
lrwxrwxrwx 1 root root 107 Feb 13 02:39 kube-scheduler-controller-0_kube-system_kube-scheduler-8bf38e5a52d643b527b00c977083bcb1ed2b11ffa328b95c20af4231e6f2426a.log -> /var/log/pods/kube-system_kube-scheduler-controller-0_fda5073439efac844d4f3e4b1aa6d3ab/kube-scheduler/1.log
lrwxrwxrwx 1 root root 96 Feb 13 02:40 kube-proxy-pxb4j_kube-system_kube-proxy-b1a5e0f841f935910e32f1ca12d27f94e5a2ac83148aace35e9269e7bf471864.log -> /var/log/pods/kube-system_kube-proxy-pxb4j_f3774ba3-47ad-4fdf-a979-318331469613/kube-proxy/1.log
lrwxrwxrwx 1 root root 126 Feb 13 02:40 cm-cert-manager-cainjector-6f8dc8f64d-f4q54_cert-manager_cert-manager-1ec02196fb9a9bd457eb28f2589a116d305c564d19eb14c1f15f05830fa06306.log -> /var/log/pods/cert-manager_cm-cert-manager-cainjector-6f8dc8f64d-f4q54_76d61c95-1dce-4116-a7e1-06448bf35792/cert-manager/1.log
lrwxrwxrwx 1 root root 123 Feb 13 02:40 cm-cert-manager-webhook-556b7d64d8-qwpks_cert-manager_cert-manager-e8d88f393c0d1a6207ee2cf9e06b92a278d8fe114fb984e3caa308636342e95b.log -> /var/log/pods/cert-manager_cm-cert-manager-webhook-556b7d64d8-qwpks_6194f082-3086-460a-93bc-91b19485a990/cert-manager/1.log
lrwxrwxrwx 1 root root 113 Feb 13 02:40 kube-sriov-cni-ds-amd64-6sj7x_kube-system_kube-sriov-cni-5b81b9dec561f31109c06648330e361cff796fdce08237b78faef921fb218006.log -> /var/log/pods/kube-system_kube-sriov-cni-ds-amd64-6sj7x_e1b0b7d3-7a2a-4292-bf40-5186866f36d2/kube-sriov-cni/1.log
lrwxrwxrwx 1 root root 109 Feb 13 02:40 source-controller-7f4bb65f88-fktsp_flux-helm_manager-098354bc0fefd5c6d5627aadac82f3e00d050f7062d7f0eabd9673a20f8e6f94.log -> /var/log/pods/flux-helm_source-controller-7f4bb65f88-fktsp_f292c724-9141-4879-ad7a-aba8956129e6/manager/1.log
Thank you,
Kris
