Elastic Agent excessive cpu / log output in kubernetes

Hello,

I am noticing some undesirable behavior when using elastic agent within kubernetes on-prem. I have a fleet managed elastic agent running in kubernetes with APM, System, and Kubernetes integrations turned on. I was able to get it working to a degree where I could see logs in kibana. What was unusual was the high cpu and excessive log output. Eventually the node just became to unstable and went down.

My goal is to send my Open Telemetry Trace that are generated from my .net 6 container apps to elasticsearch. My understanding is that my apps can send their otel logs to an elastic agent that has an apm policy. I used the generated DaemonSet yaml during the policy/agent creation in Kibana to install the elastic agent. I am using this documentation as my guide. Components and documentation | APM User Guide [8.7] | Elastic

I have a dedicated Elasticsearch cluster running version 8.7. Agent version is 8.7.

I have uploaded some logs that show 100s of logs per second that could explain the high cpu. Let me know if need more information.

{"log.level":"info","@timestamp":"2023-04-22T15:10:29.055Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.055Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.056Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.062Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-04-22T15:10:29.062Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":929},"message":"Component state changed system/metrics-default (HEALTHY->FAILED): Failed: pid '10154' exited with code '-1'","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"FAILED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-04-22T15:10:29.062Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":929},"message":"Unit state changed system/metrics-default-system/metrics-system-0fad99e6-e7bb-41de-91e3-cbbb96a5bb03 (HEALTHY->FAILED): Failed: pid '10154' exited with code '-1'","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"FAILED"},"unit":{"id":"system/metrics-default-system/metrics-system-0fad99e6-e7bb-41de-91e3-cbbb96a5bb03","type":"input","state":"FAILED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-04-22T15:10:29.062Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":929},"message":"Unit state changed system/metrics-default (HEALTHY->FAILED): Failed: pid '10154' exited with code '-1'","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"FAILED"},"unit":{"id":"system/metrics-default","type":"output","state":"FAILED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.068Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.073Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.075Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.076Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.079Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.082Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.085Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.089Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.091Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.092Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.093Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.095Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.095Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.098Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.098Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.098Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.098Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.098Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.098Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.098Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.098Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.099Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.099Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.100Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.100Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.100Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.101Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.101Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.25.0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":73,"file.name":"add_kubernetes_metadata/kubernetes.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.103Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.104Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.103Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.106Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.111Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.115Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.115Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.117Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.117Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.117Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.117Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.117Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.117Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.119Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.120Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.120Z","message":"kubernetes: Node docker-desktop discovered by machine-id matching","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"kubernetes","log.origin":{"file.line":138,"file.name":"kubernetes/util.go"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.121Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.122Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.122Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.122Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-22T15:10:29.122Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":102,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-22T15:10:29.122Z","message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":81,"file.name":"add_cloud_metadata/provider_aws_ec2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Observing the same. Also, tried to set agent.logging.level to "warning" or "error" but elastic-agents still print out "info" messages.

This should be fixed in 8.7.1 based on the log messages you are seeing. [Elastic Agent] Default processors created per input can result in high agent CPU usage · Issue #35000 · elastic/beats · GitHub

The fix is in Beats so the changelog entry is there: Beats version 8.7.1 | Beats Platform Reference [8.7] | Elastic

  • Beats started by agent now configure one global instance of each default add_x_metadata processor instead of creating one processor per input. Fixes a significant performance degradation for the aws-s3 input. 35000 35031

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.