Kubernetes agent with self signed certificate

Elastic Observability
1

Hi,

I have installed an on-prem elastic cluster with kibana and fleet.
All use xpack with a self signed root certificate.

I installed elastic agents for logging and metrics on multiple servers (Ubuntu 20.04) where the root certifcate is added to the trusted root certificates, which work fine.

Now for the kubernetes cluster i spend few hours trying to get them working but cannot find a solution. All nodes have the root certificate added to the trusted root certificates, but i keep getting untrusted certificate error and get no data in elastic.

When i shell into the pod and look at filebeat logging, i see the following error:

{"log.level":"error","@timestamp":"2022-08-18T19:42:55.209Z","log.logger":"esclientleg","log.origin":{"file.name":"transport/logging.go","file.line":38},"message":"Error dialing x509: certificate signed by unknown authority","service.name":"filebeat","network":"tcp","address":"elastic-001.xxxx:9200","ecs.version":"1.6.0"}

I also added the root certificate as a kubernetes secret, mounted it into the containers and used env vars to configure the certificate. The documentation about the env vars is not 100% clear to me.

Here is part of the agent deploy manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elastic-agent
  namespace: kube-system
  labels:
    app: elastic-agent
spec:
  selector:
    matchLabels:
      app: elastic-agent
  template:
    metadata:
      labels:
        app: elastic-agent
    spec:
      ...
      containers:
        - name: elastic-agent
          image: docker.elastic.co/beats/elastic-agent:8.3.3
          env:
            - name: FLEET_ENROLL
              value: "1"
            # Set to true in case of insecure or unverified HTTP
            - name: FLEET_INSECURE
              value: "true"
              # The ip:port pair of fleet server
            - name: FLEET_URL
              value: "https://elastic-fleet.xxxx:8220"
              # If left empty KIBANA_HOST, KIBANA_FLEET_USERNAME, KIBANA_FLEET_PASSWORD are needed
            - name: FLEET_ENROLLMENT_TOKEN
              value: "xxxxx"
            - name: FLEET_SERVER_ELASTICSEARCH_CA
              value: /etc/pki/elastic-ca.crt
            - name: ELASTICSEARCH_CA
              value: /etc/pki/elastic-ca.crt
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          ...
          volumeMounts:
            ...
            - name: elastic-ca
              mountPath: /etc/pki/elastic-ca.crt
              subPath: elastic-ca.crt
              readOnly: true
      volumes:
        ...
        - name: elastic-ca
          secret:
            secretName: elastic-ca

Any documentation on how to configure self signed certificates for elastic agent on kubernetes?
The agents do show as healthy in fleet, so that part works.

Kind regards,
Randy

2

After more trying i found a working solution.

I have mounted the custom CA directly into the /etc/ssl/certs folder and removed the env vars for CA.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elastic-agent
  namespace: kube-system
  labels:
    app: elastic-agent
spec:
  selector:
    matchLabels:
      app: elastic-agent
  template:
    metadata:
      labels:
        app: elastic-agent
    spec:
      ...
      containers:
        - name: elastic-agent
          image: docker.elastic.co/beats/elastic-agent:8.3.3
          env:
            - name: FLEET_ENROLL
              value: "1"
            # Set to true in case of insecure or unverified HTTP
            - name: FLEET_INSECURE
              value: "true"
              # The ip:port pair of fleet server
            - name: FLEET_URL
              value: "https://elastic-fleet.xxxx:8220"
              # If left empty KIBANA_HOST, KIBANA_FLEET_USERNAME, KIBANA_FLEET_PASSWORD are needed
            - name: FLEET_ENROLLMENT_TOKEN
              value: "xxxxx"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          ...
          volumeMounts:
            ...
            - name: elastic-ca
              mountPath: /etc/ssl/certs/elastic-ca.crt
              subPath: elastic-ca.crt
              readOnly: true
      volumes:
        ...
        - name: elastic-ca
          secret:
            secretName: elastic-ca