Any better solution to make elastic agent DaemonSet collect log smoothly?

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
, ,
4

I deleted the posts because it has many mistakes.

Now, I solved my first question

When i want to use RollingFile , i have to create PVC to share log to elastic agent?

The answer is yes, i have to create PV to share data to elastic agent. Otherwise, elastic agent can not find log files.

This is my manifest:

Part of elasticsearch.yaml

  nodeSets:
    - name: coordinate
      count: 1
      config:
        node.roles: []
        path.logs: /var/log/elasticsearch/
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 2Gi
            storageClassName: standard
        - metadata:
            name: logging-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
            storageClassName: standard
      podTemplate:
        spec:
          initContainers:
            - name: grant-logging-permission
              securityContext:
                privileged: true
                runAsUser: 0
              command:
                [
                  "sh",
                  "-c",
                  "chgrp 1000 /var/log/elasticsearch/ && chmod g+rwx /var/log/elasticsearch/ && ls -l /var/log/ ",
                ]
              volumeMounts:
                - name: logging-data
                  mountPath: /var/log/elasticsearch/
          containers:
            - name: elasticsearch
              env:
                - name: PRE_STOP_ADDITIONAL_WAIT_SECONDS
                  value: "5"
                - name: READINESS_PROBE_TIMEOUT
                  value: "60"
              resources:
                requests:
                  memory: 2Gi
                  cpu: 200m
                limits:
                  memory: 2Gi
              volumeMounts:
                - name: logging-config
                  mountPath: /usr/share/elasticsearch/config/log4j2.properties
                  subPath: log4j2.properties
                  readOnly: true
                - name: logging-data
                  mountPath: /var/log/elasticsearch/
          volumes:
            - name: logging-config
              configMap:
                name: elasticsearch-logging-config
            - name: logging-data
              emptyDir: {}

Agent.yaml

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: elastic-system
spec:
  version: 8.8.0
  kibanaRef:
    name: kibana-cluster
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        hostNetwork: true
        # hostPID: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: agent
            resources:
              limits:
                cpu: 2000m
                memory: 4Gi
            volumeMounts:
              - name: varlogpods
                mountPath: /var/log/pods
                readOnly: true
              - name: logging-pv
                mountPath: /var/log/pv
                readOnly: true
        volumes:
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: logging-pv
            hostPath:
              path: /tmp/hostpath-provisioner/elastic-system/

But i have question when i am writing volumeClaimTemplates

  • How to control PV hostpath, when nodeSets.count more than 1? /tmp/hostpath-provisioner/elastic-system/ contains elasticsearch data folders, i dont want to share them to elastic agent.