# Collect container logs with elastic-agent

**URL:** <https://discuss.elastic.co/t/collect-container-logs-with-elastic-agent/352935>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [February 9, 2024, 12:56pm UTC](https://discuss.elastic.co/t/collect-container-logs-with-elastic-agent/352935 "2024-02-09T12:56:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![lduvnjak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lduvnjak/32/77724_2.png) [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\
**Post date:** [February 9, 2024, 12:56pm UTC](https://discuss.elastic.co/t/collect-container-logs-with-elastic-agent/352935/1 "2024-02-09T12:56:19Z")

</div>

Hey Everyone,

I'm having some trouble getting the Kubernetes integration to fully work on my ECK cluster.

What I'm trying to do is get elastic-agent to read container logs and forward them to ES.

Here's my elastic-agent.yaml:

```auto
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: default
spec:
  version: 8.12.0
  kibanaRef:
    name: kibana
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        tolerations:
        - key: "node-role.kubernetes.io/control-plane"
          operator: "Exists"
          effect: "NoSchedule"
        serviceAccountName: elastic-agent
        hostNetwork: true
        hostPID: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: default
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: default
  name: elastic-agent
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: default
roleRef:
  kind: Role
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: elastic-agent-kubeadm-config
  namespace: default
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: default
roleRef:
  kind: Role
  name: elastic-agent-kubeadm-config
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - namespaces
      - events
      - pods
      - services
      - configmaps
      - serviceaccounts
      - persistentvolumes
      - persistentvolumeclaims
    verbs: ["get", "list", "watch"]
  - apiGroups: ["extensions"]
    resources:
      - replicasets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - statefulsets
      - deployments
      - replicasets
      - daemonsets
    verbs: ["get", "list", "watch"]
  - apiGroups:
      - ""
    resources:
      - nodes/stats
    verbs:
      - get
  - apiGroups: ["batch"]
    resources:
      - jobs
      - cronjobs
    verbs: ["get", "list", "watch"]
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources:
      - clusterrolebindings
      - clusterroles
      - rolebindings
      - roles
    verbs: ["get", "list", "watch"]
  - apiGroups: ["policy"]
    resources:
      - podsecuritypolicies
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources:
      - storageclasses
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elastic-agent
  namespace: default
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elastic-agent-kubeadm-config
  namespace: default
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups: [""]
    resources:
      - configmaps
    resourceNames:
      - kubeadm-config
    verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: default
  labels:
    k8s-app: elastic-agent

```

I've tried messing around with `volumeMounts` but when trying to specify them alongside `volumes` I get the following error:  
`Warning ReconciliationError 42m (x23 over 108m) agent-controller Reconciliation error: DaemonSet.apps "elastic-agent-agent" is invalid: spec.template.spec.containers[0].image: Required value`

The volumes and volumeMounts can be found [here](https://raw.githubusercontent.com/elastic/elastic-agent/8.12/deploy/kubernetes/elastic-agent-managed-kubernetes.yaml). The issue is that it's referenced from these [docs](https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-managed-by-fleet.html#_step_1_download_the_elastic_agent_manifest), and in these docs elastic-agent is run as a DaemonSet directly, not through the [Agent CRD](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-elastic-agent-fleet.html) that comes with ECK operator.

Any help is greatly apprecited!

Cheers,  
Luka

---

<div class="post-metadata">

**Author:** ![lduvnjak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lduvnjak/32/77724_2.png) [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\
**Post date:** [February 9, 2024, 1:00pm UTC](https://discuss.elastic.co/t/collect-container-logs-with-elastic-agent/352935/2 "2024-02-09T13:00:01Z")

</div>

Here's the yaml I was working with that threw the error above

```auto
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: default
spec:
  version: 8.12.0
  kibanaRef:
    name: kibana
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        tolerations:
        - key: "node-role.kubernetes.io/control-plane"
          operator: "Exists"
          effect: "NoSchedule"
        serviceAccountName: elastic-agent
        hostNetwork: true
        hostPID: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: elastic-agent
            volumeMounts:
              - name: proc
                mountPath: /hostfs/proc
                readOnly: true
              - name: cgroup
                mountPath: /hostfs/sys/fs/cgroup
                readOnly: true
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
                readOnly: true
              - name: varlog
                mountPath: /var/log
                readOnly: true
              - name: etc-full
                mountPath: /hostfs/etc
                readOnly: true
              - name: var-lib
                mountPath: /hostfs/var/lib
                readOnly: true
              - name: etc-mid
                mountPath: /etc/machine-id
                readOnly: true
              - name: sys-kernel-debug
                mountPath: /sys/kernel/debug
              - name: elastic-agent-state
                mountPath: /usr/share/elastic-agent/state
        volumes:
          - name: agent-data
            emptyDir: {}
          - name: proc
            hostPath:
              path: /proc
          - name: cgroup
            hostPath:
              path: /sys/fs/cgroup
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers
          - name: varlog
            hostPath:
              path: /var/log
          - name: etc-full
            hostPath:
              path: /etc
          - name: var-lib
            hostPath:
              path: /var/lib
          - name: etc-mid
            hostPath:
              path: /etc/machine-id
              type: File
          - name: sys-kernel-debug
            hostPath:
              path: /sys/kernel/debug
          - name: elastic-agent-state
            hostPath:
              path: /var/lib/elastic-agent-managed/default/state
              type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: default
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: default
  name: elastic-agent
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: default
roleRef:
  kind: Role
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: elastic-agent-kubeadm-config
  namespace: default
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: default
roleRef:
  kind: Role
  name: elastic-agent-kubeadm-config
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - namespaces
      - events
      - pods
      - services
      - configmaps
      - serviceaccounts
      - persistentvolumes
      - persistentvolumeclaims
    verbs: ["get", "list", "watch"]
  - apiGroups: ["extensions"]
    resources:
      - replicasets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - statefulsets
      - deployments
      - replicasets
      - daemonsets
    verbs: ["get", "list", "watch"]
  - apiGroups:
      - ""
    resources:
      - nodes/stats
    verbs:
      - get
  - apiGroups: ["batch"]
    resources:
      - jobs
      - cronjobs
    verbs: ["get", "list", "watch"]
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources:
      - clusterrolebindings
      - clusterroles
      - rolebindings
      - roles
    verbs: ["get", "list", "watch"]
  - apiGroups: ["policy"]
    resources:
      - podsecuritypolicies
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources:
      - storageclasses
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elastic-agent
  namespace: default
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elastic-agent-kubeadm-config
  namespace: default
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups: [""]
    resources:
      - configmaps
    resourceNames:
      - kubeadm-config
    verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: default
  labels:
    k8s-app: elastic-agent

```

The reference I used was from [this](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/4) post where he hasn't specified the image but apparently it works.

**EDIT** - even when copying [this](https://raw.githubusercontent.com/elastic/cloud-on-k8s/2.11/config/recipes/elastic-agent/fleet-custom-logs-integration.yaml) example I get the same error

---

<div class="post-metadata">

**Author:** ![lduvnjak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lduvnjak/32/77724_2.png) [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\
**Post date:** [February 9, 2024, 1:53pm UTC](https://discuss.elastic.co/t/collect-container-logs-with-elastic-agent/352935/3 "2024-02-09T13:53:35Z")

</div>

One more thing I've noticed is when applying a new configuration it recreates the pods, and all data is lost (aka there's a massive amount of **Offline** agents in the fleet UI). Even though by default data should persist across restarts:

`kubectl describe po elastic-agent-agent-97mjg`

```auto
Volumes:
  agent-data:
    Type: HostPath (bare host directory volume)
    Path: /var/lib/elastic-agent/default/elastic-agent/state
    HostPathType: DirectoryOrCreate

```

There's also a massive amount of these errors (for every module):  
`{"log.level":"error","@timestamp":"2024-02-09T13:57:49.055Z","message":"Failed to list light metricsets for module postgresql: getting metricsets for module 'postgresql': loading light module 'postgresql' definition: loading module configuration from '/usr/share/elastic-agent/data/elastic-agent-3c8be7/components/module/postgresql/module.yml': config file (\"/usr/share/elastic-agent/data/elastic-agent-3c8be7/components/module/postgresql/module.yml\") must be owned by the user identifier (uid=0) or root","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":145,"file.name":"mb/lightmodules.go"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"registry.lightmodules","ecs.version":"1.6.0"}`

Seems to be related to this [issue](https://github.com/elastic/cloud-on-k8s/issues/6266). The module files are indeed owned by `elastic-agent` while the container itself is running as `root`. Can't find any errors related to why it doesn't remember it's state though. HostPath seems to be mounted accordingly and it's present on the nodes:

```auto
[root@k8s-master01 ~]# ll /var/lib/elastic-agent/default/elastic-agent/state
total 0
drwxr-x---. 5 root root 75 Feb 9 15:43 data

```

---

<div class="post-metadata">

**Author:** ![lduvnjak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lduvnjak/32/77724_2.png) [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\
**Post date:** [February 9, 2024, 3:27pm UTC](https://discuss.elastic.co/t/collect-container-logs-with-elastic-agent/352935/4 "2024-02-09T15:27:08Z")

</div>

Managed to get it working. Posting here in case anyone needs it. I have no clue what I've changed to get it to work tbh, it was trial and error

```auto
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: default
spec:
  version: 8.12.0
  kibanaRef:
    name: kibana
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        tolerations:
        - key: "node-role.kubernetes.io/control-plane"
          operator: "Exists"
          effect: "NoSchedule"
        serviceAccountName: elastic-agent
        hostNetwork: true
        hostPID: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        containers:
        - name: agent
          volumeMounts:
            - name: proc
              mountPath: /hostfs/proc
              readOnly: true
            - name: cgroup
              mountPath: /hostfs/sys/fs/cgroup
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: etc-full
              mountPath: /hostfs/etc
              readOnly: true
            - name: var-lib
              mountPath: /hostfs/var/lib
              readOnly: true
            - name: etc-mid
              mountPath: /etc/machine-id
              readOnly: true
            - name: sys-kernel-debug
              mountPath: /sys/kernel/debug
        securityContext:
          runAsUser: 0
        volumes:
          - name: proc
            hostPath:
              path: /proc
          - name: cgroup
            hostPath:
              path: /sys/fs/cgroup
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers
          - name: varlog
            hostPath:
              path: /var/log
          - name: etc-full
            hostPath:
              path: /etc
          - name: var-lib
            hostPath:
              path: /var/lib
          - name: etc-mid
            hostPath:
              path: /etc/machine-id
              type: File
          - name: sys-kernel-debug
            hostPath:
              path: /sys/kernel/debug

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 8, 2024, 3:27pm UTC](https://discuss.elastic.co/t/collect-container-logs-with-elastic-agent/352935/5 "2024-03-08T15:27:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.