Hi. I am currently working on deploying Fleet + Elastic Agent in a test-ish environment. I've had success in enrolling the agents into Fleet as follows: sudo ./elastic-agent install --url=https://fleet1.local:8220 --enrollment-token=[REDACTED] --certificate-authorities /etc/elastic-agent/certs/elastic-stack-ca.pem
However, I saw option --ca-sha256, which ostensibly could be used to validate the Elasticsearch / Fleet certificates (both are signed with the same CA without any intermediate CAs), and I am itching to use it, so that I would not have to keep the certificates on the agent hosts (one less thing to manage). However, whenever I'm trying to enroll an agent with this option (with or without the --certificate-authorities key), the output looks like this:
[user@agent1 elastic-agent-8.0.1-linux-x86_64]$ sudo ./elastic-agent install --url=https://fleet1.local:8220 --enrollment-token=[REDACTED] --ca-sha256=[REDACTED]
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2022-12-01T10:55:03.149-0500","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":456},"message":"Starting enrollment to URL: https://fleet1.local:8220/","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.0/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
Initially I assumed that this was due to the Fleet server not presenting the CA certificate due to it not being present in the chain (openssl s_client -showcerts -verify 5 -status -connect fleet.local:8220 < /dev/null returns: Verify return code: 21 (unable to verify the first certificate)). To fix that, I passed the whole certificate chain to the Fleet server (cat fleet-cert.pem elastic-ca.pem > fleet-cert-chain.pem), which seems to fix things as far as openssl is concerned:
[user@fleet.local certs]$ openssl s_client -showcerts -verify 5 -status -connect fleet.local:8220 < /dev/null
verify depth is 5
CONNECTED(00000003)
OCSP response: no response sent
depth=1 CN = Elastic Certificate Tool Autogenerated CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 CN = Elastic Certificate Tool Autogenerated CA
verify return:1
depth=0 CN = fleet.local
verify return:1
---
Certificate chain
0 s:/CN=fleet.local
i:/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
##REDACTED BLOCK 1##
-----END CERTIFICATE-----
1 s:/CN=Elastic Certificate Tool Autogenerated CA
i:/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
##REDACTED BLOCK 2##
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=fleet.local
issuer=/CN=Elastic Certificate Tool Autogenerated CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
##SKIPPED##
Verify return code: 19 (self signed certificate in certificate chain)
---
DONE
[user@fleet.local certs]$
[user@fleet.local certs]$ cat << EOF | base64 --decode | sha256sum
##REDACTED BLOCK 2##
EOF
0e15######################REDACTED#########################293b2 -
[user@fleet.local certs]$ head -n -1 /etc/elastic-agent/certs/elastic-stack-ca.pem | tail -n +2 | base64 --decode | sha256sum
0e15######################REDACTED#########################293b2 -
The CA fingerprint presented by openssl above matches the one in my CA certificate (and also the one presented by my Kibana web interface, as reported by my browser), but I'm still having the same error. Does anyone know why that might be the case? --fleet-server-es-ca-trusted-fingerprint worked just fine during Fleet server deployment.
The --fleet-server-es-ca-trusted-fingerprint and --ca-sha256 options do fundamentally different things and is likely the source of your confusion.
The --fleet-server-es-ca-trusted-fingerprint option will trust any certificate signed by the CA (cert) with the specified fingerprint. In this case the CA does not need to be passed or specified to the agent.
The --ca-sha256 option is used to version pin a (CA) certificate that's in the system store, or specified by another option.
If you want to just provide a CA fingerprint that the agent trusts without deploying the cert, the --fleet-server-es-ca-trusted-fingerprint options is the one you want to use.
We have encountered the same issue described by @username11 , i.e. our use case is avoiding having to use a file with a self-signed CA's public certificate in PEM format during Elastic Agent enrollment. We expected --ca-sha256 would have a similar effect to using ca_trusted_fingerprint when configuring Beats but this does not seem to be the case.
We have tried using --fleet-server-es-ca-trusted-fingerprint as suggested but this results in the same "Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority" error.
Note that we are not trying to enroll the first Elastic Agent into a Fleet Server policy but are trying to enroll subsequent Agents that are connecting to an existing Fleet Server whose certificate is signed using a self-signed CA.
We see several issues mostly pertaining to the documentation:
The encroll command's synopsis suggets that --fleet-server-es-ca-trusted-fingerprint can only be used when enrolling an Elastic Agent into a Fleet Server policy. For normal enrollments only --ca-sha256 is available.
--ca-sha256's description is the same as used in enroll.go source code, i.e. "Comma separated list of certificate authorities hash pins used for certificate verifications" without any additional explanations provided:
2.1. It is not obvious that the SHA256 hash requires a CA certificate that must be made available by other means as you have suggested, e.g. by having it stored in the system keystore or by specifying it as an option.
2.2. There's no instructions on what the format of the string representing the fingerprint should be (this holds true for --fleet-server-es-ca-trusted-fingerprint as well). Several options come to mind: openssl outputs the SHA256 hash in hexadecimal with colon as a separator. The ca_trusted_fingerprint Beats configuration directive expects the colons to be removed. The hexadecimal numbers A-F might perhaps be required to be specified as lowercase letters. Alternatively Base64-encoding might be required. An example would be best to clear up any confusion. Metricbeat documentation for ca_trusted_fingerprint also provides a shell snippet using openssl to extract the fingerprint in the proper format.
Configure SSL/TLS for self-managed Fleet Servers | Fleet and Elastic Agent Guide [8.5] | Elastic only covers specifying a certificate fingerprint for connecting to Elasticsearch, not to Fleet Server (the format of the string is omitted again). The last section "Install your Elastic Agents and enroll them in Fleet" only specifies a certificate file via --certificate-authorities=/path/to/ca.crt leading us to suspect that as of Elastic Agent 8.5 it is actually not possible to enroll the Agent without specifying a PEM file.
We have encountered the same issue described by @username11 , i.e. our use case is avoiding having to use a file with a self-signed CA's public certificate in PEM format during Elastic Agent enrollment. We expected --ca-sha256 would have a similar effect to using ca_trusted_fingerprint when configuring Beats but this does not seem to be the case.
Correct, as I stated before they do different things (pinning vs trust)
We have tried using --fleet-server-es-ca-trusted-fingerprint as suggested but this results in the same "Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority " error.
That's expected, the --fleet-server-es-ca-trusted-fingerprint option is for the fleet-server instance to trust an elasticsearch instance. An agent (that does not run a server) does not use it.
The encroll command's synopsis suggets that --fleet-server-es-ca-trusted-fingerprint can only be used when enrolling an Elastic Agent into a Fleet Server policy. For normal enrollments only --ca-sha256 is available.
That's a correct summary, the --fleet-server-* options are only used when the agent is running an instance of fleet.
--ca-sha256 's description is the same as used in enroll.go source code, i.e. "Comma separated list of certificate authorities hash pins used for certificate verifications" without any additional explanations provided:
Very good point, we can indeed improve our documentation to help with all of these cases.
Configure SSL/TLS for self-managed Fleet Servers | Fleet and Elastic Agent Guide [8.5] | Elastic only covers specifying a certificate fingerprint for connecting to Elasticsearch, not to Fleet Server (the format of the string is omitted again). The last section "Install your Elastic Agents and enroll them in Fleet" only specifies a certificate file via --certificate-authorities=/path/to/ca.crt leading us to suspect that as of Elastic Agent 8.5 it is actually not possible to enroll the Agent without specifying a PEM file.
Once again correct. Currently there is no option that would do the same for elastic-agent to fleet-server. It should be an open issue in the elastic-agent repo; but we do not have a timeline for implementation.
You'll have to deploy the pem CA to your agent nodes in order to enrol them.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.