# Any better solution to make elastic agent DaemonSet collect log smoothly?

**URL:** <https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Tags:** elastic-stack-monitoring, docker, fleet\
**Created:** [May 31, 2023, 7:56am UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763 "2023-05-31T07:56:30Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kelvin\_Chan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kelvin_chan/32/119470_2.png) [@Kelvin\_Chan](https://discuss.elastic.co/u/Kelvin_Chan)\
**Post date:** [May 31, 2023, 7:56am UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/1 "2023-05-31T07:56:30Z")

</div>

I deployed a elastic agent DaemonSet to monitor elasticsearch cluster, but elastic agent mounts `/var/log/containers`, `/var/log/pods`, `/var/lib/docker/containers` to read container log, but these log files are only contain console log. If i modified `log4j2.properties` and changed to `RollingFile`, those paths become useless.

> When i want to use `RollingFile`, i have to create PVC to share log to elastic agent?

Or

> I should use `RollingFile` and `Console` mode at the same time? But all kinds of logs mix in one log file whether elastic agent can handle it properly?

This is my agent manifest

```yml
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: elastic-system
spec:
  version: 8.8.0
  kibanaRef:
    name: kibana-cluster
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        hostNetwork: true
        # hostPID: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: agent
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
                readOnly: true
              - name: varlogpods
                mountPath: /var/log/pods
                readOnly: true
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
                readOnly: true
        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers

```

---

<div class="post-metadata">

**Author:** ![Kelvin\_Chan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kelvin_chan/32/119470_2.png) [@Kelvin\_Chan](https://discuss.elastic.co/u/Kelvin_Chan)\
**Post date:** [June 1, 2023, 5:51am UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/4 "2023-06-01T05:51:07Z")

</div>

I deleted the posts because it has many mistakes.

Now, I solved my first question

> When i want to use `RollingFile` , i have to create PVC to share log to elastic agent?

The answer is _yes_, i have to create PV to share data to elastic agent. Otherwise, elastic agent can not find log files.

This is my manifest:

Part of elasticsearch.yaml

```yml
  nodeSets:
    - name: coordinate
      count: 1
      config:
        node.roles: []
        path.logs: /var/log/elasticsearch/
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 2Gi
            storageClassName: standard
        - metadata:
            name: logging-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
            storageClassName: standard
      podTemplate:
        spec:
          initContainers:
            - name: grant-logging-permission
              securityContext:
                privileged: true
                runAsUser: 0
              command:
                [
                  "sh",
                  "-c",
                  "chgrp 1000 /var/log/elasticsearch/ && chmod g+rwx /var/log/elasticsearch/ && ls -l /var/log/ ",
                ]
              volumeMounts:
                - name: logging-data
                  mountPath: /var/log/elasticsearch/
          containers:
            - name: elasticsearch
              env:
                - name: PRE_STOP_ADDITIONAL_WAIT_SECONDS
                  value: "5"
                - name: READINESS_PROBE_TIMEOUT
                  value: "60"
              resources:
                requests:
                  memory: 2Gi
                  cpu: 200m
                limits:
                  memory: 2Gi
              volumeMounts:
                - name: logging-config
                  mountPath: /usr/share/elasticsearch/config/log4j2.properties
                  subPath: log4j2.properties
                  readOnly: true
                - name: logging-data
                  mountPath: /var/log/elasticsearch/
          volumes:
            - name: logging-config
              configMap:
                name: elasticsearch-logging-config
            - name: logging-data
              emptyDir: {}

```

Agent.yaml

```yml
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: elastic-system
spec:
  version: 8.8.0
  kibanaRef:
    name: kibana-cluster
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        hostNetwork: true
        # hostPID: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: agent
            resources:
              limits:
                cpu: 2000m
                memory: 4Gi
            volumeMounts:
              - name: varlogpods
                mountPath: /var/log/pods
                readOnly: true
              - name: logging-pv
                mountPath: /var/log/pv
                readOnly: true
        volumes:
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: logging-pv
            hostPath:
              path: /tmp/hostpath-provisioner/elastic-system/

```

But i have question when i am writing `volumeClaimTemplates`

- How to control PV hostpath, when `nodeSets.count` more than 1? `/tmp/hostpath-provisioner/elastic-system/` contains elasticsearch data folders, i dont want to share them to elastic agent.

---

<div class="post-metadata">

**Author:** ![Kelvin\_Chan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kelvin_chan/32/119470_2.png) [@Kelvin\_Chan](https://discuss.elastic.co/u/Kelvin_Chan)\
**Post date:** [June 1, 2023, 10:14am UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/5 "2023-06-01T10:14:48Z")

</div>

The remaining question at last post is happened in kibana. Since i deploy two kibana and use PV to share log to elastic agent, two kibana containers have to use same log file. I can't image what will happen when rolling file.

```yml
  podTemplate:
    spec:
      initContainers:
        - name: grant-logging-permission
          securityContext:
            privileged: true
            runAsUser: 0
          command:
            [
              "sh",
              "-c",
              "chgrp 1000 /var/log/kibana/ && chmod g+rwx /var/log/kibana/ && ls -l /var/log/",
            ]
          volumeMounts:
            - name: kibana-log
              mountPath: /var/log/kibana/
      containers:
        - name: kibana
          env:
            - name: NODE_OPTIONS
              value: "--max-old-space-size=2048"
          resources:
            requests:
              memory: 1Gi
              cpu: 0.5
            limits:
              memory: 4Gi
              cpu: 4
          volumeMounts:
            - name: kibana-log
              mountPath: /var/log/kibana/
      volumes:
        - name: kibana-log
          persistentVolumeClaim:
            claimName: logging-data-kibana-cluster # two kibana using same PV

```

---

<div class="post-metadata">

**Author:** ![Kelvin\_Chan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kelvin_chan/32/119470_2.png) [@Kelvin\_Chan](https://discuss.elastic.co/u/Kelvin_Chan)\
**Post date:** [June 1, 2023, 12:23pm UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/6 "2023-06-01T12:23:32Z")

</div>

K8s is using json format to record console output line by line, i have to create new pipeline to process data. The elastic stack log must be recorded in json format, otherwise the multi-line log problem needs to be solved.

I dont think i can write a pipeline can handle this:

```auto
cat /var/lib/docker/containers/1b3a8d98ac1a521666514f2412da326ae3c1a4ded76a779f7cc4c4bbb314d241/1b3a8d98ac1a521666514f2412da326ae3c1a4ded76a779f7cc4c4bbb314d241-json.log

"log":"\u0009Root causes:\n","stream":"stdout","time":"2023-06-01T11:26:44.1790961Z"}
{"log":"\u0009\u0009index_not_found_exception: no such index [logs]\n","stream":"stdout","time":"2023-06-01T11:26:44.1791021Z"}
{"log":" at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791066Z"}
{"log":" at runMicrotasks (\u003canonymous\u003e)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791109Z"}
{"log":" at processTicksAndRejections (node:internal/process/task_queues:96:5)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791152Z"}
{"log":" at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791199Z"}
{"log":" at Indices.resolveIndex (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:1059:16)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791246Z"}
{"log":" at /usr/share/kibana/node_modules/@kbn/data-view-management-plugin/server/routes/resolve_index.js:29:18\n","stream":"stdout","time":"2023-06-01T11:26:44.179129Z"}
{"log":" at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:149:30)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791335Z"}
{"log":" at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:115:50)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791377Z"}
{"log":" at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791562Z"}
{"log":" at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791604Z"}
{"log":" at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791639Z"}
{"log":" at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791689Z"}
{"log":" at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791728Z"}
{"log":"[2023-06-01T11:26:44.653+00:00][ERROR][http] ResponseError: index_not_found_exception\n","stream":"stdout","time":"2023-06-01T11:26:44.6548179Z"}
{"log":"\u0009Root causes:\n","stream":"stdout","time":"2023-06-01T11:26:44.6548599Z"}
{"log":"\u0009\u0009index_not_found_exception: no such index [logs-]\n","stream":"stdout","time":"2023-06-01T11:26:44.6548692Z"}

```

---

<div class="post-metadata">

**Author:** ![Kelvin\_Chan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kelvin_chan/32/119470_2.png) [@Kelvin\_Chan](https://discuss.elastic.co/u/Kelvin_Chan)\
**Post date:** [June 2, 2023, 11:06am UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/7 "2023-06-02T11:06:23Z")

</div>

1. RollingFile and each nodes have their own persistent volume.
  - pros:
    - easy to implement
    - independent volumes and log files

  - cons:
    - too many persistent volume
    - Some services(e.g. kibana) can't do this when node count more than one

2. RollingFile, but sharing one persistent volume.
  - pros:
    - the easiest way
    - available for all types of service
    - only need one persistent volume

  - cons:
    - multiple nodes concurrently write data to same file and dont know what will happen while rolling to next file

3. Console, collecting log from kubernetes console file
  - pros:
    - do not need persistent volume

  - cons:
    - need to add one more pipeline to unwrap k8s console json data e.g. logstash
    - mixing all kinds of log into one file

I decide to use the second plan, and set rolling size to 1GB.

---

<div class="post-metadata">

**Author:** ![Kelvin\_Chan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kelvin_chan/32/119470_2.png) [@Kelvin\_Chan](https://discuss.elastic.co/u/Kelvin_Chan)\
**Post date:** [June 4, 2023, 2:27am UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/9 "2023-06-04T02:27:09Z")

</div>

Whether i can enable buffer and increase buffer size to reduce concurrently writing?

[https://logging.apache.org/log4j/2.x/manual/appenders.html#FileAppender](https://logging.apache.org/log4j/2.x/manual/appenders.html#FileAppender)

---

<div class="post-metadata">

**Author:** ![Kelvin\_Chan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kelvin_chan/32/119470_2.png) [@Kelvin\_Chan](https://discuss.elastic.co/u/Kelvin_Chan)\
**Post date:** [June 4, 2023, 7:44am UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/10 "2023-06-04T07:44:52Z")

</div>

Use second plan then disable `immediateFlush` and set a smaller `bufferSize` e.g. 2kb. This is my best solution.

---

<div class="post-metadata">

**Author:** ![Kelvin\_Chan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kelvin_chan/32/119470_2.png) [@Kelvin\_Chan](https://discuss.elastic.co/u/Kelvin_Chan)\
**Post date:** [June 18, 2023, 12:23pm UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/11 "2023-06-18T12:23:07Z")

</div>

I think i found a better solution, thanks for kubernetes users. Temp volume is a better choice, if combine that with topology i think i will get ideal result.

---

<div class="post-metadata">

**Author:** ![Kelvin\_Chan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kelvin_chan/32/119470_2.png) [@Kelvin\_Chan](https://discuss.elastic.co/u/Kelvin_Chan)\
**Post date:** [June 18, 2023, 12:41pm UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/12 "2023-06-18T12:41:42Z")

</div>

Here is part of my manifest

```yml
                  subPath: log4j2.properties
                - name: temp-logging
                  mountPath: /var/log/elasticsearch/
          volumes:
            - name: logging-config
              configMap:
                name: elasticsearch-logging-config
            - name: temp-logging
              hostPath:
                path: /tmp/logging/elasticsearch/
                type: Directory

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 16, 2023, 12:41pm UTC](https://discuss.elastic.co/t/any-better-solution-to-make-elastic-agent-daemonset-collect-log-smoothly/334763/13 "2023-07-16T12:41:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.