I have eck installed in a fresh kubernetes cluster and i am attempting to fetch container logs
version: 8.15.2
spec:
elasticsearchRefs:
- name: elasticsearch-prod-eck-elasticsearch
kibanaRefs:
- name: kibana-prod-eck-kibana
daemonSet:
podTemplate:
spec:
automountServiceAccountToken: true
serviceAccountName: elastic-agent
containers:
- name: agent
securityContext:
runAsUser: 0
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
config:
agent:
enabled: true
logs: true
metrics: true
inputs:
- id: container-log-$${kubernetes.pod.name}-$${kubernetes.container.id}
type: filestream
use_output: default
meta:
package:
name: kubernetes
version: 1.52.0
data_stream:
namespace: default
streams:
- id: container-log-$${kubernetes.pod.name}-$${kubernetes.container.id}
data_stream:
dataset: kubernetes.container_logs
type: logs
prospector.scanner.symlinks: true
parsers:
- container: ~
paths:
- /var/log/containers/*$${kubernetes.container.id}.log
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
providers:
kubernetes:
node: $${NODE_NAME}
scope: node
hints:
enabled: true
default_container_logs: true
I assumed providers.kubernetes.hints.default_container_logs would collect the logs for me but it hasn't.
Since then i have added volume and volume mounts, and i have also added config.inputs.
No container logs are being collected.
What am i doing wrong?
The hints feature is enabled in the config - Hints annotations based autodiscover | Fleet and Elastic Agent Guide [8.15] | Elastic
./elastic-agent inspect --variables --variables-wait 1s -c /etc/elastic-agent/agent.yml
agent:
enabled: true
logs: true
metrics: true
outputs:
default:
hosts:
- https://elasticsearch-prod-eck-elasticsearch-es-http.elastic-system.svc:9200
password: <redacted>
ssl:
certificate_authorities:
- /mnt/elastic-internal/elasticsearch-association/elastic-system/elasticsearch-prod-eck-elasticsearch/certs/ca.crt
type: elasticsearch
username: elastic-system-agent-prod-eck-agent-elastic-system-elasticsearch-prod-eck-elasticsearch-agent-user
providers:
kubernetes:
hints:
default_container_logs: true
enabled: true
node: ip-10-1-140-8.eu-west-2.compute.internal
scope: node
i do see logs in var/log/containers which are symlinked? to pods
found some logs in /usr/share/elastic-agent/elastic-agent-20241014.ndjson
{"log.level":"warn","@timestamp":"2024-10-14T14:48:21.765Z","log.logger":"composable.providers.kubernetes.cfgwarn","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes.(*dynamicProvider).Run","file.name":"kubernetes/kubernetes.go","file.line":61},"message":"BETA: Hints' feature is beta.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:21.766Z","log.logger":"composable.providers.docker","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker.(*dynamicProvider).Run","file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:21.766Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes.(*dynamicProvider).watchResource","file.name":"kubernetes/kubernetes.go","file.line":112},"message":"Kubernetes provider started for resource pod with node scope","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:21.766Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/kubernetes.DiscoverKubernetesNode","file.name":"kubernetes/util.go","file.line":123},"message":"kubernetes: Using node ip-10-1-2-195.eu-west-2.compute.internal provided in the config","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:22.191Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes.(*dynamicProvider).watchResource","file.name":"kubernetes/kubernetes.go","file.line":112},"message":"Kubernetes provider started for resource node with node scope","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-14T14:48:22.191Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/kubernetes.DiscoverKubernetesNode","file.name":"kubernetes/util.go","file.line":123},"message":"kubernetes: Using node ip-10-1-2-195.eu-west-2.compute.internal provided in the config","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-14T14:48:23.766Z","log.logger":"composable","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable.(*controller).Run.func1","file.name":"composable/controller.go","file.line":139},"message":"failed to run provider 'kubernetes_leaderelection': context deadline exceeded","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-14T14:48:23.766Z","log.logger":"composable","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable.(*controller).Run.func1","file.name":"composable/controller.go","file.line":139},"message":"failed to run provider 'host': context deadline exceeded","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-14T14:48:23.766Z","log.logger":"composable","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable.(*controller).Run.func2","file.name":"composable/controller.go","file.line":156},"message":"failed to run provider 'kubernetes': context deadline exceeded","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-14T14:48:23.766Z","log.logger":"composable","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable.(*controller).Run.func1","file.name":"composable/controller.go","file.line":139},"message":"failed to run provider 'kubernetes_secrets': context deadline exceeded","ecs.version":"1.6.0"}