Error dialing x509: certificate signed by unknown authority Kubernetes integration

Elastic Stack Elasticsearch
1

I know there is allot on info about this, but I'm not grokking what needs to be done.
The agents for the integration with kubernetes deploys in the kube-system namespace. i am using the quickstart fyi. What do i need to do to get the cert that was generated in the eck name space to the agents in kube-system?

Error dialing x509: certificate signed by unknown authority

this is the cert i want yes?

elasticsearch-es-http-ca-internal
2

I think you need to take a look on configuration files of elasticsearch you will find the certificate on the directory and the name on elasticsearch.yml file

  • did u use self-signed certificate or public ?
    if it self signe just make the authority certificate of elasticsearch certificate trusted in the server
3

It may depend on where that error was logged and whether that error occurs during fleet enrollment or while interacting with Elasticsearch

If it's for the output to Elasticsearch the CA can be specified here

1000008492
10000084921008×1984 183 KB

If the issue is fleet enrollment you can either set FLEET_INSECURE to true or you can mount the ca certificate into the container and set FLEET_CA to grab the self signed CA

4

I Didn't create any certificates it's all just done via the operator right?

This is ECK FYI

5

Thank you again for your help, Fleet is good! It's just all the metrics reporting to elastic.

image
image795×930 63.4 KB

{"log.level":"error","@timestamp":"2024-11-21T21:45:37.018Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":39,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2"},"service.name":"metricbeat","network":"tcp","address":"elasticsearch-es-http.eck.svc:9200","ecs.version":"1.6.0","log.logger":"esclientleg","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-21T21:45:37.018Z","message":"Ping request failed with: Get \"https://elasticsearch-es-http.eck.svc:9200\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.logger":"esclientleg","log.origin":{"file.line":306,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-11-21T21:45:40.567Z","message":"Failed to connect to backoff(elasticsearch(https://elasticsearch-es-http.eck.svc:9200)): Get \"https://elasticsearch-es-http.eck.svc:9200\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-21T21:45:40.567Z","message":"Attempting to reconnect to backoff(elasticsearch(https://elasticsearch-es-http.eck.svc:9200)) with 1190 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-21T21:45:40.567Z","message":"ES Ping(url=https://elasticsearch-es-http.eck.svc:9200)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":302,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-11-21T21:45:40.577Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"address":"elasticsearch-es-http.eck.svc:9200","service.name":"metricbeat","network":"tcp","ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":39,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-21T21:45:40.577Z","message":"Ping request failed with: Get \"https://elasticsearch-es-http.eck.svc:9200\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":306,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping"},"service.name":"metricbeat","ecs.version":"1.6.0"}

I am also seeing a ca certificate in the pod/container itself. I actually deleted it out of the container deleted the deployment and reapplied thinking it might have had an old certificate or something but I don't know where that certificate is coming from/

 ls -l /etc/ssl/certs/
-rw-r--r-- 1 root root 219342 Oct 10 10:11  ca-certificates.crt
6

Yes, you can read more here:

By default, it creates a self-signed CA and issues certificates to each component.

That screenshot is just showing that you've currently got the default fleet policy configured in your ECK CRD manifest. You shared the CRD in your other post:

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: eck
spec:
  version: 8.15.3
  count: 1
  elasticsearchRef:
    name: elasticsearch
  http:
    service:
      spec:
        type: LoadBalancer
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.eck.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.eck.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        namespace: eck
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: eck
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system

You could remove the config and manage it via the GUI or you could add xpack.fleet.agents.elasticsearch.ca_sha256 to the CRD and populate it with the base64-encoded string of the SHA-256 fingerprint of the CA Cert which in ECK is called something ending with -es-http-ca-internal

7

Like this?

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: eck
spec:
  version: 8.15.3
  count: 1
  elasticsearchRef:
    name: elasticsearch
  http:
    service:
      spec:
        type: LoadBalancer
  config:
    xpack.fleet.agents.elasticsearch.ca_sha256: ["NjdiMTg1NDA0YTMwZThmNWFlYzkxMDAyYjIwNzVkYTk0N2RjOWU2Mg=="]
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.eck.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.eck.svc:8220"]
    xpack.fleet.packages:
8

Yeah, that's what I would expect assuming that's the base64 encoded version of the sha256 of the ca cert

After setting that you should see it appear in the fleet output settings as well

9

I've tried every combination of every certificate certificate fingerprint and base encoded a 64 and I cannot get it to work.

This is the steps that I've taken.

i get the cert from kubernets secret

kubectl get secrets -n eck elasticsearch-es-http-ca-internal -o json | jq -r '.data."tls.crt"' | base64 -d > ecki.crt

-----BEGIN CERTIFICATE-----
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
etc...
-----END CERTIFICATE-----

Then i get the Finger print

 openssl x509 -in ecki.crt -noout -fingerprint -sha256
SHA256 Fingerprint=47:C5:BF:5E:4F:4D:AE:DB:B6:D3:6A:DE:7E:92:1B:6E:66:08:10:1B:83:25:81:EE:80:DE:5F:0D:A7:1F:AE:31

then i remove the ":"

47C5BF5E4F4DAEDBB6D36ADE7E921B6E6608101B832581EE80DE5F0DA71FAE31

then I base64 encode it

R8W/Xk9Nrtu202refpIbbmYIEBuDJYHugN5fDacfrjE=

I put it in my yaml like so

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: eck
spec:
  version: 8.15.3
  count: 1
  elasticsearchRef:
    name: elasticsearch
  http:
    service:
      spec:
        type: LoadBalancer
  config:
    xpack.fleet.agents.elasticsearch.ca_sha256: "R8W/Xk9Nrtu202refpIbbmYIEBuDJYHugN5fDacfrjE="
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.eck.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.eck.svc:8220"]
    xpack.fleet.packages:

nothing shows up in fleet out setting

image
image766×117 7.63 KB

I am working off this Definition

xpack.fleet.agents.elasticsearch.ca_sha256
Hash pin used for certificate verification. The pin is a base64-encoded string of the SHA-256 fingerprint.

I've tried using the non hash fingerprint i've tried using the entire certificate I don't know if this is an order of operations issue because I can't get that fingerprint till the all-in-one is deployed does need something extra to grab that config like a restart or something?

To sum up my question how do i get the agents in the kube-system To talk to elasticsearch with the default created secrets done by the operator? I just wanted to be clear.

Thank you for everyone's time for reading this I appreciate it again I'm just trying to learn and understand i've been reading documentation I've been going to Github looking for any open issues and reading through manifest there I'm really trying. Obviously I'm not understanding something .

10

Could you try the workaround provided here Certificate ca_sha256 ignored in Fleet initialisation config file · Issue #139411 · elastic/kibana · GitHub

Where you define a xpack.fleet.outputs: with ca_trusted_fingerprint defined?

11

Thank you so much your post what I did just so if anyone finds this in the future it'll make more sense than it did to me.

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: eck
spec:
  version: 8.15.3
  count: 1
  elasticsearchRef:
    name: elasticsearch
  config:
#   xpack.fleet.agents.elasticsearch.ca_sha256: "R8W/Xk9Nrtu202refpIbbmYIEBuDJYHugN5fDacfrjE="
#   xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.eck.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.eck.svc:8220"]
    xpack.fleet.outputs:
      - id: fleet-default-output
        name: Default
        type: elasticsearch
        hosts: ["https://elasticsearch-es-http.eck.svc:9200"]
    # openssl x509 -fingerprint -sha256 -noout -in tls/kibana/elasticsearch-ca.pem (colons removed)
        ca_trusted_fingerprint: 47C5BF5E4F4DAEDBB6D36ADE7E921B6E6608101B832581EE80DE5F0DA71FAE31
        is_default: true
        is_default_monitoring: true
   etc....