Environment details
Kubernetes cluster RKE2 v1.27.3 with DISA STIG's
ECK Operator: 2.9.0 (Ironbank image)
Elastic Agent Image: 8.9.0 (Ironbank image)
Issue:
After getting the pod(s) fleet server and agents to a running / healthy state, I still receive the following errors from Metricbeat
{"log.level":"error","@timestamp":"2023-09-18T15:03:33.088Z","message":"Failed to connect to backoff(elasticsearch(https://elasticcluster-siem-es-http.elasticcluster-siem.svc:9200)): Get \"https://elasticcluster-siem-es-http.elasticcluster-siem.svc:9200\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-18T15:03:33.089Z","message":"Attempting to reconnect to backoff(elasticsearch(https://elasticcluster-siem-es-http.elasticcluster-siem.svc:9200)) with 196 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-18T15:03:33.099Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.origin":{"file.line":38,"file.name":"transport/logging.go"},"network":"tcp","ecs.version":"1.6.0","log.logger":"esclientleg","service.name":"metricbeat","address":"elasticcluster-siem-es-http.elasticcluster-siem.svc:9200","ecs.version":"1.6.0"}
I imagine the issue is something with self-signed certificates (Dev environment) however, I believe I've correctly added them as a trusted certificate and it should therefore work.
Steps
### Create self-signed certificate
1. create a ca.cfg file from the example
2. openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout tls.key -out tls.pem -config example-ca-siem.cfg -extensions 'v3_req'
3. cp tls.pem ca.crt && cp tls.pem tls.crt
4. kubectl create secret generic elastic-siem-cert --from-file=ca.crt --from-file=tls.crt --from-file=tls.key --namespace=elasticcluster-siem
5. kubectl create secret generic proxy-ca-secret-elastic-siem --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-file=ca.crt=ca.crt --namespace=elasticcluster-siem
6. kubectl create secret tls ingress-tls-cert-elastic-siem --cert=tls.pem --key=tls.key -n elasticcluster-siem
Example ca.cfg
### Create self-signed certificate
1. create a ca.cfg file from the example
2. openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout tls.key -out tls.pem -config example-ca-siem.cfg -extensions 'v3_req'
3. cp tls.pem ca.crt && cp tls.pem tls.crt
4. kubectl create secret generic elastic-siem-cert --from-file=ca.crt --from-file=tls.crt --from-file=tls.key --namespace=elasticcluster-siem
5. kubectl create secret generic proxy-ca-secret-elastic-siem --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-file=ca.crt=ca.crt --namespace=elasticcluster-siem
6. kubectl create secret tls ingress-tls-cert-elastic-siem --cert=tls.pem --key=tls.key -n elasticcluster-siem
Example
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = Whatever
L = Whatever
O = elasticcluster
OU = Elastic
CN = ElasticSearch SIEM
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = elastic-siem.elasticcluster-dev.gov
DNS.2 = kibana-siem.elasticcluster-dev.gov
DNS.3 = elasticcluster-siem-es-http.elasticcluster-siem.svc
verified the following variable matches the CA
FLEET_SERVER_ELASTICSEARCH_CA=/mnt/elastic-internal/elasticsearch-association/emissive-siem/emissive-siem/certs/ca.crt
Kubernetes manifest snippets
ENVs
env:
- name: FLEET_CA
value: /usr/share/fleet-server/config/http-certs/ca.crt
- name: FLEET_ENROLL
value: "true"
- name: FLEET_ENROLLMENT_TOKEN
valueFrom:
secretKeyRef:
key: FLEET_ENROLLMENT_TOKEN
name: fleet-server-siem-agent-envvars
optional: false
- name: FLEET_SERVER_CERT
value: /usr/share/fleet-server/config/http-certs/tls.crt
- name: FLEET_SERVER_CERT_KEY
value: /usr/share/fleet-server/config/http-certs/tls.key
- name: FLEET_SERVER_ELASTICSEARCH_CA
value: /mnt/elastic-internal/elasticsearch-association/elasticcluster-siem/elasticcluster-siem/certs/ca.crt
- name: FLEET_SERVER_ELASTICSEARCH_HOST
value: https://elasticcluster-siem-es-http.elasticcluster-siem.svc:9200
- name: FLEET_SERVER_ENABLE
value: "true"
- name: FLEET_SERVER_POLICY_ID
value: eck-fleet-server
- name: FLEET_SERVER_SERVICE_TOKEN
value: somevalue
- name: FLEET_URL
value: https://fleet-server-siem-agent-http.elasticcluster-siem.svc:8220
- name: CONFIG_PATH
value: /usr/share/elastic-agent
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
VolumeMounts
volumeMounts:
- mountPath: /usr/share/elastic-agent/state
name: agent-data
- mountPath: /etc/agent.yml
name: config
readOnly: true
subPath: agent.yml
- mountPath: /mnt/elastic-internal/elasticsearch-association/elasticcluster-siem/elasticcluster-siem/certs
name: elasticsearch-certs
readOnly: true
- mountPath: /usr/share/fleet-server/config/http-certs
name: fleet-certs
readOnly: true
volumes
volumes:
- emptyDir:
sizeLimit: 500Mi
name: agent-data
- name: config
secret:
defaultMode: 288
optional: false
secretName: fleet-server-siem-agent-config
- name: elasticsearch-certs
secret:
defaultMode: 420
optional: false
secretName: fleet-server-siem-agent-es-elasticcluster-siem-elasticcluster-siem-ca
- name: elasticsearch-certs-0
secret:
defaultMode: 420
optional: false
secretName: fleet-server-siem-agent-es-elasticcluster-siem-elasticcluster-siem-ca
- name: fleet-certs
secret:
defaultMode: 420
optional: false
secretName: fleet-server-siem-agent-http-certs-internal
any assistance greatly appreciated