Fleet-server and elastic-agent metricbeat x509 unknown CA on kubernetes

Environment details

Kubernetes cluster RKE2 v1.27.3 with DISA STIG's
ECK Operator: 2.9.0 (Ironbank image)
Elastic Agent Image: 8.9.0 (Ironbank image)

Issue:

After getting the pod(s) fleet server and agents to a running / healthy state, I still receive the following errors from Metricbeat

{"log.level":"error","@timestamp":"2023-09-18T15:03:33.088Z","message":"Failed to connect to backoff(elasticsearch(https://elasticcluster-siem-es-http.elasticcluster-siem.svc:9200)): Get \"https://elasticcluster-siem-es-http.elasticcluster-siem.svc:9200\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-18T15:03:33.089Z","message":"Attempting to reconnect to backoff(elasticsearch(https://elasticcluster-siem-es-http.elasticcluster-siem.svc:9200)) with 196 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2023-09-18T15:03:33.099Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.origin":{"file.line":38,"file.name":"transport/logging.go"},"network":"tcp","ecs.version":"1.6.0","log.logger":"esclientleg","service.name":"metricbeat","address":"elasticcluster-siem-es-http.elasticcluster-siem.svc:9200","ecs.version":"1.6.0"}

I imagine the issue is something with self-signed certificates (Dev environment) however, I believe I've correctly added them as a trusted certificate and it should therefore work.

Steps

### Create self-signed certificate

1. create a ca.cfg file from the example

2. openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout tls.key -out tls.pem -config example-ca-siem.cfg -extensions 'v3_req'

3. cp tls.pem ca.crt && cp tls.pem tls.crt

4. kubectl create secret generic elastic-siem-cert --from-file=ca.crt --from-file=tls.crt --from-file=tls.key --namespace=elasticcluster-siem

5. kubectl create secret generic proxy-ca-secret-elastic-siem --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-file=ca.crt=ca.crt --namespace=elasticcluster-siem

6. kubectl create secret tls ingress-tls-cert-elastic-siem --cert=tls.pem --key=tls.key -n elasticcluster-siem

Example ca.cfg

### Create self-signed certificate

1. create a ca.cfg file from the example

2. openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout tls.key -out tls.pem -config example-ca-siem.cfg -extensions 'v3_req'

3. cp tls.pem ca.crt && cp tls.pem tls.crt

4. kubectl create secret generic elastic-siem-cert --from-file=ca.crt --from-file=tls.crt --from-file=tls.key --namespace=elasticcluster-siem

5. kubectl create secret generic proxy-ca-secret-elastic-siem --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-file=ca.crt=ca.crt --namespace=elasticcluster-siem

6. kubectl create secret tls ingress-tls-cert-elastic-siem --cert=tls.pem --key=tls.key -n elasticcluster-siem


Example

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = Whatever
L = Whatever
O = elasticcluster
OU = Elastic
CN = ElasticSearch SIEM
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = elastic-siem.elasticcluster-dev.gov
DNS.2 = kibana-siem.elasticcluster-dev.gov
DNS.3 = elasticcluster-siem-es-http.elasticcluster-siem.svc

verified the following variable matches the CA

FLEET_SERVER_ELASTICSEARCH_CA=/mnt/elastic-internal/elasticsearch-association/emissive-siem/emissive-siem/certs/ca.crt

Kubernetes manifest snippets

ENVs

env:
        - name: FLEET_CA
          value: /usr/share/fleet-server/config/http-certs/ca.crt
        - name: FLEET_ENROLL
          value: "true"
        - name: FLEET_ENROLLMENT_TOKEN
          valueFrom:
            secretKeyRef:
              key: FLEET_ENROLLMENT_TOKEN
              name: fleet-server-siem-agent-envvars
              optional: false
        - name: FLEET_SERVER_CERT
          value: /usr/share/fleet-server/config/http-certs/tls.crt
        - name: FLEET_SERVER_CERT_KEY
          value: /usr/share/fleet-server/config/http-certs/tls.key
        - name: FLEET_SERVER_ELASTICSEARCH_CA
          value: /mnt/elastic-internal/elasticsearch-association/elasticcluster-siem/elasticcluster-siem/certs/ca.crt
        - name: FLEET_SERVER_ELASTICSEARCH_HOST
          value: https://elasticcluster-siem-es-http.elasticcluster-siem.svc:9200
        - name: FLEET_SERVER_ENABLE
          value: "true"
        - name: FLEET_SERVER_POLICY_ID
          value: eck-fleet-server
        - name: FLEET_SERVER_SERVICE_TOKEN
          value: somevalue
        - name: FLEET_URL
          value: https://fleet-server-siem-agent-http.elasticcluster-siem.svc:8220
        - name: CONFIG_PATH
          value: /usr/share/elastic-agent
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName

VolumeMounts

volumeMounts:
        - mountPath: /usr/share/elastic-agent/state
          name: agent-data
        - mountPath: /etc/agent.yml
          name: config
          readOnly: true
          subPath: agent.yml
        - mountPath: /mnt/elastic-internal/elasticsearch-association/elasticcluster-siem/elasticcluster-siem/certs
          name: elasticsearch-certs
          readOnly: true
        - mountPath: /usr/share/fleet-server/config/http-certs
          name: fleet-certs
          readOnly: true
		  
		  
volumes

volumes:
      - emptyDir:
          sizeLimit: 500Mi
        name: agent-data
      - name: config
        secret:
          defaultMode: 288
          optional: false
          secretName: fleet-server-siem-agent-config
      - name: elasticsearch-certs
        secret:
          defaultMode: 420
          optional: false
          secretName: fleet-server-siem-agent-es-elasticcluster-siem-elasticcluster-siem-ca
      - name: elasticsearch-certs-0
        secret:
          defaultMode: 420
          optional: false
          secretName: fleet-server-siem-agent-es-elasticcluster-siem-elasticcluster-siem-ca
      - name: fleet-certs
        secret:
          defaultMode: 420
          optional: false
          secretName: fleet-server-siem-agent-http-certs-internal

any assistance greatly appreciated

I was able to resolve, the quickstart guide has the following set

xpack.fleet.agents.elasticsearch.hosts:

remove this from the Kibana.yaml config and recreate kibana pods

once this is removed you can edit the output in Fleet within Kibana UI (Fleet -> Settings -> Outsputs -> actions )

Add the following to "Advanced YAML Configuration"

ssl.certificate_authorities: ["/mnt/elastic-internal/elasticsearch-association/elastic-siem/elastic-siem/certs/ca.crt"]

your path may differ

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.