I've used the example K8s manifest to deploy Elastic Agent on our AWS EKS cluster. I followed the documentation in the EKS section to comment out modules that are unavailable in AWS EKS.
Recently, I upgraded to 8.7.0 and noticed a huge amount of log volume increase. A large portion of this was coming from Elastic Agent logs itself. Some of this was resolved in 8.7.1 due to issues with the logging, and since then setting the logging level to warning has reduced my events per minute from about 100k to about 2k. Most of the remaining 2k Elastic Agent logs are the following:
filestream input with ID '' already exists, this will lead to data duplication, please use a different ID
I added IDs to every data_stream in my configmap, and discovered it's related to:
- type: filestream
id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
use_output: default
meta:
package:
name: kubernetes
version: 1.29.2
data_stream:
namespace: default
streams:
- data_stream:
dataset: kubernetes.container_logs
type: logs
prospector.scanner.symlinks: true
parsers:
- container: ~
# - ndjson:
# target: json
# - multiline:
# type: pattern
# pattern: '^\['
# negate: true
# match: after
paths:
- /var/log/containers/*${kubernetes.container.id}.log
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "our-cluster-name"
url: "https://our-cluser.uri.aws.com"
In combination with this, I have also been receiving the warning:
DEPRECATED: Log input. Use Filestream input instead.
Is there a working filestream configuration that could be used for a Standalone Deployment that doesn't result large volumes of the warnings described above?