8.7.1: Stand Alone Kubernetes Deployment - filestream input with ID '' already exists

I've used the example K8s manifest to deploy Elastic Agent on our AWS EKS cluster. I followed the documentation in the EKS section to comment out modules that are unavailable in AWS EKS.

Recently, I upgraded to 8.7.0 and noticed a huge amount of log volume increase. A large portion of this was coming from Elastic Agent logs itself. Some of this was resolved in 8.7.1 due to issues with the logging, and since then setting the logging level to warning has reduced my events per minute from about 100k to about 2k. Most of the remaining 2k Elastic Agent logs are the following:

filestream input with ID '' already exists, this will lead to data duplication, please use a different ID

I added IDs to every data_stream in my configmap, and discovered it's related to:

- type: filestream
        id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
        use_output: default
            name: kubernetes
            version: 1.29.2
          namespace: default
          - data_stream:
              dataset: kubernetes.container_logs
              type: logs
            prospector.scanner.symlinks: true
              - container: ~
              # - ndjson:
              #     target: json
              # - multiline:
              #     type: pattern
              #     pattern: '^\['
              #     negate: true
              #     match: after
              - /var/log/containers/*${kubernetes.container.id}.log
              - add_fields:
                  target: orchestrator.cluster
                    name: "our-cluster-name"
                    url: "https://our-cluser.uri.aws.com"

In combination with this, I have also been receiving the warning:

DEPRECATED: Log input. Use Filestream input instead.

Is there a working filestream configuration that could be used for a Standalone Deployment that doesn't result large volumes of the warnings described above?

Raised as an issue on the GitHub page for diagnostics #2701

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.