Cannot collect logs from kubernetes with containerd runtimes

Elastic Stack Elastic Agent
1

Hi everyone
We deployed elastic agents with kubernetes integrations on newly installed kubernetes clusters 1.24. We noticed some missing logs and investigated the agents logs and status: we have this kind of error for every log file

11:10:17.345
elastic_agent.filebeat
[elastic_agent.filebeat][debug] Incoming log.file.path value: /var/log/containers/elastic-agent-f5tqx_monitoring_elastic-agent-c38bb0fa3843aed8800b5bbe230984693f6b1860f4e46c6910c46a4fcbd6dbb2.log
11:10:17.345
elastic_agent.filebeat
[elastic_agent.filebeat][debug] log.file.path value does not contain matcher's logs_path '/var/lib/docker/containers/', skipping...
11:10:17.345
elastic_agent.filebeat
[elastic_agent.filebeat][debug] No container match string, not adding kubernetes data

and the detailed elastic-agent status --output yaml command returns a lot of this. one for each log file

  - unit_id: filestream-default-filestream-container-logs-00222d86-dbb3-401d-8d4b-c00aeddc4fc1-kubernetes-40aada03-ef38-4dde-a6e9-68e1d06dd132.api-server
    unit_type: 0
    state: 4
    message: 'Failed: pid ''11842'' exited with code ''-1'''

Since kubernetes 1.24 uses containerd and not docker anymore, how should we configure the kubernetes integration? are those filebeat crashes related to the missing /var/lib/docker folder?

I tried every combination of processor configuration but seems it's been overridden by the main filebeat configuration which I have not accesso to, being within elastic agent.
can you help me please?

2

Hi @venturieffect,
Is this the same issue as Elastic Agent 8.6.x standalone deployment in Kubernetes doesn't start monitoring new pods until agent restart · Issue #2269 · elastic/elastic-agent · GitHub?

3

Hi Michel
It may be the case, yes.
We noticed two separate issues:

  • On one kubernetes cluster the filestream units are constantly crashing as showed in my opening post
  • on basically all other kubernetes clusters the filestream units are healty but the count is way less than the actual number of pods in the node. I also experienced what your github issue says: if the pods are restarted, they now appears among the filestream units
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.