Elastic-agent: Error dialing x509: certificate signed by unknown authority

[enigmatic]

Hi All,

Hope you can help wit the following.

I installed Elasticsearch and Kibana version 8.4. both are up and running.
I'm trying to install Fleet to manage the elastic-agents, how ever i hit a road block.

the elastic-agent shows as healthy in the Fleet dashboard, how ever no logs are coming in.
During my analysis i checked the log files:
/opt/Elastic/Agent/data/elastic-agent-d3eb3e/logs/default/metricbeat-20221003-2.ndjson
and
/opt/Elastic/Agent/data/elastic-agent-d3eb3e/logs/default/filebeat-20221003-2.ndjson
i find:

Error dialing x509: certificate signed by unknown authority

during the setup of Fleet i created the certificates to use. the es-ca certificate was auto-generated.

when i check the es-ca end point via:

curl --cacert /etc/pki/elasticsearch/elasticsearch-ca.crt -u elastic https://machine-host-name:9200

i get correct output:

{
  "name" : "machine-host-name",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "T7Ed8T7BQUm0pIXOFBxTlw",
  "version" : {
    "number" : "8.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "89f8c6d8429db93b816403ee75e5c270b43a940a",
    "build_date" : "2022-09-14T16:26:04.382547801Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

the command i use to enroll the Fleetserver is:

/elastic-agent install --url=https://ip-adress-of-other-server:8220 
--fleet-server-es=https://machine-host-name:9200 
--fleet-server-service-token=token 
--fleet-server-policy=fleet-server-policy 
--fleet-server-es-ca=/etc/pki/elasticsearch/elasticsearch-ca.crt 
--certificate-authorities=/etc/pki/elasticsearch/ca.crt 
--fleet-server-cert=/etc/pki/elasticsearch/fleet-server.crt 
--fleet-server-cert-key=/etc/pki/elasticsearch/fleet-server.key

where elasticsearch-ca.crt is the same as generated by elasticsearch on setup,
location =

/etc/elasticsearch/certs/http_ca.crt

what am i doing wrong? I'm kinda lost here.
any help will be greatly appreciated!

[enigmatic]

update:
i checked to see what the CN of the certificate is that was autogenerated.
so i did the following:

openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -out cert.crt -clcerts -nokeys

and then:

openssl x509 -noout -subject -in cert.crt

this gives this name:

subject=CN = Elasticsearch security auto-configuration HTTP CA

not sure if this can be related to the issue..

[enigmatic]

update 2:
to verify if the certifcate that i'm using is correct i executed the following commands:

curl --cacert /etc/pki/elasticsearch/elasticsearch-ca.crt -u elastic https://machine-host-name:9200

as also stated above, this works. i also removed the --cacerts flag with following result:

curl -u elastic https://machine-host-name:9200
Enter host password for user 'elastic':
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

to me this suggest that the certificate is correct, so why is my Fleet agent saying it isn't?

i also updated my enroll command with the --fleet-server-es-insecure flag. according to the documentation this should override the error given. command is now:

./elastic-agent install --url=https://ip-adress-of-other-server:8220 
--fleet-server-es=https://machine-host-name:9200 
--fleet-server-service-token=token 
--fleet-server-policy=fleet-server-policy 
--certificate-authorities=/etc/pki/elasticsearch/ca.crt 
--fleet-server-cert=/etc/pki/elasticsearch/fleet-server.crt 
--fleet-server-cert-key=/etc/pki/elasticsearch/fleet-server.key 
--fleet-server-es-insecure

however the error is still present:

{"log.level":"error","@timestamp":"2022-10-03T13:19:52.043+0200","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://hud-mon-l01:9200)): Get \"https://machine-host-name:9200\": x509: certificate signed by unknown authority","service.name":"metricbeat","ecs.version":"1.6.0"

[enigmatic]

moved topic to Beats part as per github request

[jsoriano (Jaime Soriano)]

Hey @enigmatic, welcome to discuss

In case you haven't already, please take a look to the guide about using custom TLS certificates with Fleet: Configure SSL/TLS for self-managed Fleet Servers | Fleet and Elastic Agent Guide [8.4] | Elastic

Do you see the certificate error in the logs of the elastic agent running fleet server? Or this happens when trying to enroll an additional agent?

You may need to add the CA in Kibana, in the "Advanced YAML configuration" in "Management > Fleet > Settings", as described in this section: Configure SSL/TLS for self-managed Fleet Servers | Fleet and Elastic Agent Guide [8.4] | Elastic.

[shi]

@enigmatic

w ever i hit a road block.

We are also facing the same issue. So we tried with an older version.(8.3) In this older version we are not facing any issue.

We have seen that in the old version (8.3) while installing the fleet server/agent there is an additional option [ --fleet-server-es-ca-trusted-fingerprint=fingerprint_here] in the kibana agent creation dashboard.