Elastic Agent Not Collecting Kubernetes Container Logs

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I have eck installed in a fresh kubernetes cluster and i am attempting to fetch container logs

    version: 8.15.2
    spec:
      elasticsearchRefs:
      - name: elasticsearch-prod-eck-elasticsearch
      kibanaRefs:
      - name: kibana-prod-eck-kibana
      daemonSet:
        podTemplate:
          spec:
            automountServiceAccountToken: true
            serviceAccountName: elastic-agent
            containers:
            - name: agent
              securityContext:
                runAsUser: 0
              env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
              volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
            volumes:
            - name: varlogcontainers
              hostPath:
                path: /var/log/containers
            - name: varlibdockercontainers
              hostPath:
                path: /var/lib/docker/containers
    
      config:
        agent:
          enabled: true
          logs: true
          metrics: true
        inputs:
          - id: container-log-$${kubernetes.pod.name}-$${kubernetes.container.id}
            type: filestream
            use_output: default
            meta:
              package:
                name: kubernetes
                version: 1.52.0
            data_stream:
              namespace: default
            streams:
              - id: container-log-$${kubernetes.pod.name}-$${kubernetes.container.id}
                data_stream:
                  dataset: kubernetes.container_logs
                  type: logs
                prospector.scanner.symlinks: true
                parsers:
                  - container: ~
                paths:
                  - /var/log/containers/*$${kubernetes.container.id}.log
            processors:
              - add_cloud_metadata: {}
              - add_host_metadata: {}
        providers:
          kubernetes:
            node: $${NODE_NAME}
            scope: node
            hints:
              enabled: true
              default_container_logs: true

I assumed providers.kubernetes.hints.default_container_logs would collect the logs for me but it hasn't.

Since then i have added volume and volume mounts, and i have also added config.inputs.

No container logs are being collected.

What am i doing wrong?

The hints feature is enabled in the config - Hints annotations based autodiscover | Fleet and Elastic Agent Guide [8.15] | Elastic

./elastic-agent inspect --variables --variables-wait 1s -c /etc/elastic-agent/agent.yml

agent:
  enabled: true
  logs: true
  metrics: true
outputs:
  default:
    hosts:
    - https://elasticsearch-prod-eck-elasticsearch-es-http.elastic-system.svc:9200
    password: <redacted>
    ssl:
      certificate_authorities:
      - /mnt/elastic-internal/elasticsearch-association/elastic-system/elasticsearch-prod-eck-elasticsearch/certs/ca.crt
    type: elasticsearch
    username: elastic-system-agent-prod-eck-agent-elastic-system-elasticsearch-prod-eck-elasticsearch-agent-user
providers:
  kubernetes:
    hints:
      default_container_logs: true
      enabled: true
    node: ip-10-1-140-8.eu-west-2.compute.internal
    scope: node

i do see logs in var/log/containers which are symlinked? to pods

image
image1282×808 274 KB

found some logs in /usr/share/elastic-agent/elastic-agent-20241014.ndjson

{"log.level":"warn","@timestamp":"2024-10-14T14:48:21.765Z","log.logger":"composable.providers.kubernetes.cfgwarn","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes.(*dynamicProvider).Run","file.name":"kubernetes/kubernetes.go","file.line":61},"message":"BETA: Hints' feature is beta.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:21.766Z","log.logger":"composable.providers.docker","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker.(*dynamicProvider).Run","file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:21.766Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes.(*dynamicProvider).watchResource","file.name":"kubernetes/kubernetes.go","file.line":112},"message":"Kubernetes provider started for resource pod with node scope","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:21.766Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/kubernetes.DiscoverKubernetesNode","file.name":"kubernetes/util.go","file.line":123},"message":"kubernetes: Using node ip-10-1-2-195.eu-west-2.compute.internal provided in the config","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:22.191Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes.(*dynamicProvider).watchResource","file.name":"kubernetes/kubernetes.go","file.line":112},"message":"Kubernetes provider started for resource node with node scope","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:22.191Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/kubernetes.DiscoverKubernetesNode","file.name":"kubernetes/util.go","file.line":123},"message":"kubernetes: Using node ip-10-1-2-195.eu-west-2.compute.internal provided in the config","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-14T14:48:23.766Z","log.logger":"composable","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable.(*controller).Run.func1","file.name":"composable/controller.go","file.line":139},"message":"failed to run provider 'kubernetes_leaderelection': context deadline exceeded","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-14T14:48:23.766Z","log.logger":"composable","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable.(*controller).Run.func1","file.name":"composable/controller.go","file.line":139},"message":"failed to run provider 'host': context deadline exceeded","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-14T14:48:23.766Z","log.logger":"composable","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable.(*controller).Run.func2","file.name":"composable/controller.go","file.line":156},"message":"failed to run provider 'kubernetes': context deadline exceeded","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-14T14:48:23.766Z","log.logger":"composable","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable.(*controller).Run.func1","file.name":"composable/controller.go","file.line":139},"message":"failed to run provider 'kubernetes_secrets': context deadline exceeded","ecs.version":"1.6.0"}