Any better solution to make elastic agent DaemonSet collect log smoothly?

Kelvin_Chan · May 31, 2023, 7:56am

I deployed a elastic agent DaemonSet to monitor elasticsearch cluster, but elastic agent mounts /var/log/containers, /var/log/pods, /var/lib/docker/containers to read container log, but these log files are only contain console log. If i modified log4j2.properties and changed to RollingFile, those paths become useless.

When i want to use RollingFile, i have to create PVC to share log to elastic agent?

Or

I should use RollingFile and Console mode at the same time? But all kinds of logs mix in one log file whether elastic agent can handle it properly?

This is my agent manifest

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: elastic-system
spec:
  version: 8.8.0
  kibanaRef:
    name: kibana-cluster
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        hostNetwork: true
        # hostPID: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: agent
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
                readOnly: true
              - name: varlogpods
                mountPath: /var/log/pods
                readOnly: true
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
                readOnly: true
        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers

Kelvin_Chan · June 1, 2023, 5:51am

I deleted the posts because it has many mistakes.

Now, I solved my first question

When i want to use RollingFile , i have to create PVC to share log to elastic agent?

The answer is yes, i have to create PV to share data to elastic agent. Otherwise, elastic agent can not find log files.

This is my manifest:

Part of elasticsearch.yaml

  nodeSets:
    - name: coordinate
      count: 1
      config:
        node.roles: []
        path.logs: /var/log/elasticsearch/
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 2Gi
            storageClassName: standard
        - metadata:
            name: logging-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
            storageClassName: standard
      podTemplate:
        spec:
          initContainers:
            - name: grant-logging-permission
              securityContext:
                privileged: true
                runAsUser: 0
              command:
                [
                  "sh",
                  "-c",
                  "chgrp 1000 /var/log/elasticsearch/ && chmod g+rwx /var/log/elasticsearch/ && ls -l /var/log/ ",
                ]
              volumeMounts:
                - name: logging-data
                  mountPath: /var/log/elasticsearch/
          containers:
            - name: elasticsearch
              env:
                - name: PRE_STOP_ADDITIONAL_WAIT_SECONDS
                  value: "5"
                - name: READINESS_PROBE_TIMEOUT
                  value: "60"
              resources:
                requests:
                  memory: 2Gi
                  cpu: 200m
                limits:
                  memory: 2Gi
              volumeMounts:
                - name: logging-config
                  mountPath: /usr/share/elasticsearch/config/log4j2.properties
                  subPath: log4j2.properties
                  readOnly: true
                - name: logging-data
                  mountPath: /var/log/elasticsearch/
          volumes:
            - name: logging-config
              configMap:
                name: elasticsearch-logging-config
            - name: logging-data
              emptyDir: {}

Agent.yaml

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: elastic-system
spec:
  version: 8.8.0
  kibanaRef:
    name: kibana-cluster
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        hostNetwork: true
        # hostPID: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: agent
            resources:
              limits:
                cpu: 2000m
                memory: 4Gi
            volumeMounts:
              - name: varlogpods
                mountPath: /var/log/pods
                readOnly: true
              - name: logging-pv
                mountPath: /var/log/pv
                readOnly: true
        volumes:
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: logging-pv
            hostPath:
              path: /tmp/hostpath-provisioner/elastic-system/

But i have question when i am writing volumeClaimTemplates

How to control PV hostpath, when nodeSets.count more than 1? /tmp/hostpath-provisioner/elastic-system/ contains elasticsearch data folders, i dont want to share them to elastic agent.

Kelvin_Chan · June 1, 2023, 10:14am

The remaining question at last post is happened in kibana. Since i deploy two kibana and use PV to share log to elastic agent, two kibana containers have to use same log file. I can't image what will happen when rolling file.

  podTemplate:
    spec:
      initContainers:
        - name: grant-logging-permission
          securityContext:
            privileged: true
            runAsUser: 0
          command:
            [
              "sh",
              "-c",
              "chgrp 1000 /var/log/kibana/ && chmod g+rwx /var/log/kibana/ && ls -l /var/log/",
            ]
          volumeMounts:
            - name: kibana-log
              mountPath: /var/log/kibana/
      containers:
        - name: kibana
          env:
            - name: NODE_OPTIONS
              value: "--max-old-space-size=2048"
          resources:
            requests:
              memory: 1Gi
              cpu: 0.5
            limits:
              memory: 4Gi
              cpu: 4
          volumeMounts:
            - name: kibana-log
              mountPath: /var/log/kibana/
      volumes:
        - name: kibana-log
          persistentVolumeClaim:
            claimName: logging-data-kibana-cluster # two kibana using same PV

Kelvin_Chan · June 1, 2023, 12:23pm

K8s is using json format to record console output line by line, i have to create new pipeline to process data. The elastic stack log must be recorded in json format, otherwise the multi-line log problem needs to be solved.

I dont think i can write a pipeline can handle this:

cat /var/lib/docker/containers/1b3a8d98ac1a521666514f2412da326ae3c1a4ded76a779f7cc4c4bbb314d241/1b3a8d98ac1a521666514f2412da326ae3c1a4ded76a779f7cc4c4bbb314d241-json.log

"log":"\u0009Root causes:\n","stream":"stdout","time":"2023-06-01T11:26:44.1790961Z"}
{"log":"\u0009\u0009index_not_found_exception: no such index [logs]\n","stream":"stdout","time":"2023-06-01T11:26:44.1791021Z"}
{"log":"    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791066Z"}
{"log":"    at runMicrotasks (\u003canonymous\u003e)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791109Z"}
{"log":"    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791152Z"}
{"log":"    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791199Z"}
{"log":"    at Indices.resolveIndex (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:1059:16)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791246Z"}
{"log":"    at /usr/share/kibana/node_modules/@kbn/data-view-management-plugin/server/routes/resolve_index.js:29:18\n","stream":"stdout","time":"2023-06-01T11:26:44.179129Z"}
{"log":"    at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:149:30)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791335Z"}
{"log":"    at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:115:50)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791377Z"}
{"log":"    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791562Z"}
{"log":"    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791604Z"}
{"log":"    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791639Z"}
{"log":"    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791689Z"}
{"log":"    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)\n","stream":"stdout","time":"2023-06-01T11:26:44.1791728Z"}
{"log":"[2023-06-01T11:26:44.653+00:00][ERROR][http] ResponseError: index_not_found_exception\n","stream":"stdout","time":"2023-06-01T11:26:44.6548179Z"}
{"log":"\u0009Root causes:\n","stream":"stdout","time":"2023-06-01T11:26:44.6548599Z"}
{"log":"\u0009\u0009index_not_found_exception: no such index [logs-]\n","stream":"stdout","time":"2023-06-01T11:26:44.6548692Z"}

Kelvin_Chan · June 2, 2023, 11:06am

RollingFile and each nodes have their own persistent volume.
- pros:
  - easy to implement
  - independent volumes and log files
- cons:
  - too many persistent volume
  - Some services(e.g. kibana) can't do this when node count more than one
RollingFile, but sharing one persistent volume.
- pros:
  - the easiest way
  - available for all types of service
  - only need one persistent volume
- cons:
  - multiple nodes concurrently write data to same file and dont know what will happen while rolling to next file
Console, collecting log from kubernetes console file
- pros:
  - do not need persistent volume
- cons:
  - need to add one more pipeline to unwrap k8s console json data e.g. logstash
  - mixing all kinds of log into one file

I decide to use the second plan, and set rolling size to 1GB.

Kelvin_Chan · June 4, 2023, 2:27am

Whether i can enable buffer and increase buffer size to reduce concurrently writing?

https://logging.apache.org/log4j/2.x/manual/appenders.html#FileAppender

Kelvin_Chan · June 4, 2023, 7:44am

Use second plan then disable immediateFlush and set a smaller bufferSize e.g. 2kb. This is my best solution.

Kelvin_Chan · June 18, 2023, 12:23pm

I think i found a better solution, thanks for kubernetes users. Temp volume is a better choice, if combine that with topology i think i will get ideal result.

Kelvin_Chan · June 18, 2023, 12:41pm

Here is part of my manifest

                  subPath: log4j2.properties
                - name: temp-logging
                  mountPath: /var/log/elasticsearch/
          volumes:
            - name: logging-config
              configMap:
                name: elasticsearch-logging-config
            - name: temp-logging
              hostPath:
                path: /tmp/logging/elasticsearch/
                type: Directory

system · July 16, 2023, 12:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.