# Filestream Autodiscover Kubernetes duplication error logs being generated

**URL:** <https://discuss.elastic.co/t/filestream-autodiscover-kubernetes-duplication-error-logs-being-generated/353313>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 14, 2024, 9:49pm UTC](https://discuss.elastic.co/t/filestream-autodiscover-kubernetes-duplication-error-logs-being-generated/353313 "2024-02-14T21:49:19Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\
**Post date:** [February 14, 2024, 9:49pm UTC](https://discuss.elastic.co/t/filestream-autodiscover-kubernetes-duplication-error-logs-being-generated/353313/1 "2024-02-14T21:49:19Z")

</div>

Hi

We are using ELK 8.11.1 we are seeing these types of errors logs being generated from filebeat.

`filestream input with ID <> already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input.`

```auto
2024-02-14T21:40:35.840910409Z stderr F {"log.level":"error","@timestamp":"2024-02-14T21:40:35.840Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":183},"message":"filestream input with ID 'kubernetes-container-logs-mon-metricbeat-wv68v-978c73fbd06bfead737e1a56ffe8705accc188317ed9e15d0fa3fef275839e83' already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input.","service.name":"filebeat","ecs.version":"1.6.0"}
2024-02-14T21:40:35.84478493Z stderr F {"log.level":"error","@timestamp":"2024-02-14T21:40:35.844Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":183},"message":"filestream input with ID 'kubernetes-container-logs-mon-metricbeat-wv68v-aa21559a0a537be343a4488d48cfb508eb3201be22989e980a1f03f186d2e322' already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input.","service.name":"filebeat","ecs.version":"1.6.0"}
2024-02-14T21:40:35.848644233Z stderr F {"log.level":"error","@timestamp":"2024-02-14T21:40:35.848Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":183},"message":"filestream input with ID 'kubernetes-container-logs-mon-metricbeat-wv68v-788bfb798144f391e7ea1e702b691b71211a4fa1456e883b61031aa8e07b0225' already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input.","service.name":"filebeat","ecs.version":"1.6.0"}

```

This is our config

```auto
filebeat.autodiscover:
  providers:
  - add_resource_metadata:
      cronjob: false
      deployment: false
    hints.default_config:
      close.on_state_change.renamed: true
      id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
      parsers:
      - container: null
      paths:
      - /var/log/containers/*-${data.kubernetes.container.id}.log
      prospector.scanner:
        fingerprint.enabled: true
        symlinks: true
      type: filestream
    hints.enabled: true
    host: ${NODE_NAME}
    type: kubernetes
filebeat.inputs:
- close.reader.after_interval: 5m
  file_identity.fingerprint: null
  id: wra-filestream-id
  paths:
  - /var/log/*.log
  - /var/log/messages
  - /var/log/syslog
  - /var/log/**/*.log
  prospector.scanner.exclude_files:
  - ^/var/log/containers/
  - ^/var/log/pods/
  prospector.scanner.fingerprint:
    enabled: true
    length: 280
    offset: 0
  type: filestream

```

Our /var/log/containers files are symlinks of the pod's logs (/var/log/pods/\*)

```auto
ls -rtl /var/log/containers
total 448
lrwxrwxrwx 1 root root 125 Feb 13 02:39 kube-controller-manager-controller-0_kube-system_kube-controller-manager-b2b7024cdcaed8277f81936c466043d797219ff1d65bed2822755287f7ffdbbc.log -> /var/log/pods/kube-system_kube-controller-manager-controller-0_79fcaab94d97fd191c9e53535451309b/kube-controller-manager/1.log
lrwxrwxrwx 1 root root 107 Feb 13 02:39 kube-scheduler-controller-0_kube-system_kube-scheduler-8bf38e5a52d643b527b00c977083bcb1ed2b11ffa328b95c20af4231e6f2426a.log -> /var/log/pods/kube-system_kube-scheduler-controller-0_fda5073439efac844d4f3e4b1aa6d3ab/kube-scheduler/1.log
lrwxrwxrwx 1 root root 96 Feb 13 02:40 kube-proxy-pxb4j_kube-system_kube-proxy-b1a5e0f841f935910e32f1ca12d27f94e5a2ac83148aace35e9269e7bf471864.log -> /var/log/pods/kube-system_kube-proxy-pxb4j_f3774ba3-47ad-4fdf-a979-318331469613/kube-proxy/1.log
lrwxrwxrwx 1 root root 126 Feb 13 02:40 cm-cert-manager-cainjector-6f8dc8f64d-f4q54_cert-manager_cert-manager-1ec02196fb9a9bd457eb28f2589a116d305c564d19eb14c1f15f05830fa06306.log -> /var/log/pods/cert-manager_cm-cert-manager-cainjector-6f8dc8f64d-f4q54_76d61c95-1dce-4116-a7e1-06448bf35792/cert-manager/1.log
lrwxrwxrwx 1 root root 123 Feb 13 02:40 cm-cert-manager-webhook-556b7d64d8-qwpks_cert-manager_cert-manager-e8d88f393c0d1a6207ee2cf9e06b92a278d8fe114fb984e3caa308636342e95b.log -> /var/log/pods/cert-manager_cm-cert-manager-webhook-556b7d64d8-qwpks_6194f082-3086-460a-93bc-91b19485a990/cert-manager/1.log
lrwxrwxrwx 1 root root 113 Feb 13 02:40 kube-sriov-cni-ds-amd64-6sj7x_kube-system_kube-sriov-cni-5b81b9dec561f31109c06648330e361cff796fdce08237b78faef921fb218006.log -> /var/log/pods/kube-system_kube-sriov-cni-ds-amd64-6sj7x_e1b0b7d3-7a2a-4292-bf40-5186866f36d2/kube-sriov-cni/1.log
lrwxrwxrwx 1 root root 109 Feb 13 02:40 source-controller-7f4bb65f88-fktsp_flux-helm_manager-098354bc0fefd5c6d5627aadac82f3e00d050f7062d7f0eabd9673a20f8e6f94.log -> /var/log/pods/flux-helm_source-controller-7f4bb65f88-fktsp_f292c724-9141-4879-ad7a-aba8956129e6/manager/1.log

```

Thank you,  
Kris

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 14, 2024, 10:18pm UTC](https://discuss.elastic.co/t/filestream-autodiscover-kubernetes-duplication-error-logs-being-generated/353313/2 "2024-02-14T22:18:14Z")

</div>

> [@kbujold\_wr](#):
>
> ```auto
> paths: <--- THIS
> - /var/log/containers/*-${data.kubernetes.container.id}.log
> 
> ```

> [@kbujold\_wr](#):
>
> ```auto
> paths:
> - /var/log/*.log
> - /var/log/messages
> - /var/log/syslog
> - /var/log/**/*.log <---- THIS
> 
> ```

I think those are scanning the same files, perhaps I am missing something...

---

<div class="post-metadata">

**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\
**Post date:** [February 15, 2024, 2:23am UTC](https://discuss.elastic.co/t/filestream-autodiscover-kubernetes-duplication-error-logs-being-generated/353313/3 "2024-02-15T02:23:20Z")

</div>

> [@kbujold\_wr](#):
>
> ```auto
> prospector.scanner.exclude_files:
> - ^/var/log/containers/
> - ^/var/log/pods/
> 
> ```

This excludes them. Or used to before we switched from log to filestream.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 15, 2024, 4:46am UTC](https://discuss.elastic.co/t/filestream-autodiscover-kubernetes-duplication-error-logs-being-generated/353313/4 "2024-02-15T04:46:06Z")

</div>

Ahh yes missed that...

Hmmmm...

Disable the second input and see if the issue goes away...

And then debug the 2nd

And perhaps read this section closely on [simlinks](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filestream-rotating-logs)

---

<div class="post-metadata">

**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\
**Post date:** [February 15, 2024, 2:50pm UTC](https://discuss.elastic.co/t/filestream-autodiscover-kubernetes-duplication-error-logs-being-generated/353313/5 "2024-02-15T14:50:33Z")

</div>

I can carry on debugging I was looking to confirm the config syntax. Especially the `prospector.scanner.exclude_files`.

The use of the kubernetes /var/log/containers is for the metadata. As described here [filestream input | Filebeat Reference [8.12] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_prospector_scanner_symlinks).

We have use this config with no issues for many releases until we switched to filestream and moved to ELK 8.9.0. Then after this switch these errors were seen. Elastic appears to be working properly as you can see from the Discover page but these errors logs are problematic.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fc9e7ded1558ff44d74e8e2be35bb5a52d3e3802.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/1/b1076f08c083c7ee3cda01ece163674854a427c4.png)

Thanks,  
Kris

---

<div class="post-metadata">

**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\
**Post date:** [February 15, 2024, 4:30pm UTC](https://discuss.elastic.co/t/filestream-autodiscover-kubernetes-duplication-error-logs-being-generated/353313/6 "2024-02-15T16:30:50Z")

</div>

There appears to be a bug open on this ?[`filestream` input logs an error when an existing input is reloaded with the same ID · Issue #31767 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/31767)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 14, 2024, 6:31pm UTC](https://discuss.elastic.co/t/filestream-autodiscover-kubernetes-duplication-error-logs-being-generated/353313/7 "2024-03-14T18:31:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.