`
> The SHA-256 fingerprint (hash) of the certificate authority used to self-sign Elasticsearch certificates. This fingerprint will be used to verify self-signed certificates presented by Fleet Server and any inputs started by Elastic Agent for communication. This flag is required when using self-signed certificates with Elasticsearch.
When I get a chance I will look
If you generate the command from Kibana you will see (yes I know you need more ... just double checking)
```auto
sudo ./elastic-agent install \
--fleet-server-es=https://192.168.2.107:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MTk2ODUxMjIxNjI6SWt1c1NFdFRUWC1DNnB2T0xOd0w0UQ \
--fleet-server-policy=fleet-server-policy \
--fleet-server-es-ca-trusted-fingerprint=1f5fd789732ce83a08778ca577502769f8159beecc1a8d5f1edc95422321e528 \
--fleet-server-port=8220
```
BTW I did a quick metricbeat test using the trusted fingerprint and it worked... using mine I will try with yours...
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [June 29, 2024, 8:01pm UTC](https://discuss.elastic.co/t/elastic-agent-ca-trusted-fingerprint-does-not-working-but-the-certificate-works/362239/5 "2024-06-29T20:01:27Z")
> [@stephenb](#):
>
> So what is in the repo is the working version without using the fingerprint confirm?
No, there is no working version using the fingerprint, this is what I'm trying to troubleshoot.
The docker compose in the repository will perform the following steps:
- Spin-up a one-time container using the ubuntu image to create a custom CA and a certificate, this certificate will be used by the elasticsearch, kibana and fleet containers.
The certificates is stored on a persistent path that is shared with the other containers.
- After that, a Elasticsearch single-node container will be started and use the certificate generate in the previous step for both the transport and http endpoints.
- When the Elasticsearch container is healthy, a one-time container will run to setup the password for the `kibana_system` user as we cannot use the internal `elastic` user anymore.
- The next step is to spin-up a Kibana container and when it is healthy another one-time container will be executed to setup two fleet settings, the fleet host and the default output.
- This `setup_fleet` container is required to avoid the need to manually create a fleet host and to change the default output to point to the elasticsearch container, the default value is `https://localhost:9200` which of course will not work, it is in this step that the `ca_trusted_fingerprint` is used.
- The last step is to start a container running the Elastic Agent with the Fleet Server integration.
The only issue I'm facing is that the container running the Fleet Server cannot send data to the cluster because it is not recognizing the `ca_trusted_fingerprint`.
After running the compose, this is how the output configuration looks like:
If I use `openssl` to check the generate certificate, I will have this:
> sha256 Fingerprint= **8F:08:33:C6:AC:54:B9:8C:95:67:AC:57:62:14:1A:05:11:EE:63:30:B0:5E:3D:E7:7E:06:6E:2D:38:E6:FE:E4**
> -----BEGIN CERTIFICATE-----
> MIIDlzCCAn+gAwIBAgIUKnZdiBZHMD7lz90tGQBgnbG8HQEwDQYJKoZIhvcNAQEL
> BQAwWzELMAkGA1UEBhMCQlIxCzAJBgNVBAgMAlJKMRcwFQYDVQQHDA5SSU8gREUg
> SkFORUlSTzEXMBUGA1UECgwORE9DS0VSIEVMQVNUSUMxDTALBgNVBAsMBFNJRU0w
> HhcNMjQwNjI5MTkzMjQ4WhcNMjkwNzAzMTkzMjQ4WjBbMQswCQYDVQQGEwJCUjEL
> MAkGA1UECAwCUkoxFzAVBgNVBAcMDlJJTyBERSBKQU5FSVJPMRcwFQYDVQQKDA5E
> T0NLRVIgRUxBU1RJQzENMAsGA1UECwwEU0lFTTCCASIwDQYJKoZIhvcNAQEBBQAD
> ggEPADCCAQoCggEBAKX/Wj0JnJUoxzBpxyRJoCzbaT/Wf6DauwJVoKenyf7m3vba
> Q/Uysb7sOIQZ4im1IHTgFdFc9ITQtT1o7yjBmGhUxCU2VH6RxBPgLpPs8O3Hxaxi
> er0kBhB7vW9xYw9vlPGUBp5x4LtpTsCK7Qkl6im+ZCy/We5n1nvr81p0YpJFm5gG
> zUfAsG3MWHL+SFOQTN1J9vnSiL8U16gxu6/2k6KkkY8Cx49JKhH6vAWTJNYs9Jlw
> zj6LgLoBwIy2ML/AuoZ1InA/AfArGl+bbWaDIbke3JpPvbd+fjQWLpb0IJrHVC5g
> Qm/yONfX+q15Z4CYp1wiTNmo6SMfmMJiCa7RrvcCAwEAAaNTMFEwHQYDVR0OBBYE
> FCG/Y+8BqUdc7xN3bKw8oNQvvl4fMB8GA1UdIwQYMBaAFCG/Y+8BqUdc7xN3bKw8
> oNQvvl4fMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAF+wJb0x
> sEW1S/jNphRTkfNmSRS4vrrbG2Fe6uHBs1uBhyB4uDEdmY785lnXgoohtQTMV9Ux
> JW8+ydUucZ0VIpqc+w4TZo+e6/P0sJklujvQxwTaoRWeKjNTptkfYxnHp2D0T0nW
> GPPDA//GgAgNfLoYh5aAF8w4ofCIGRm+IkvCN2tF5MARxjP/ogylvwqyHBQ+KtN9
> J+1tz4BxsaB1oW3NnrI7MigJSFNAuDlraTY8cyp7mLJLeJUNFx4M+HuUQaRvyUNK
> DxCFgP3bW6MrljJj3rMNaUlKq7HoSlYv73NdMoRPz62mpp8D7eLyhq8+r+wr46Tq
> 6SokdUdWcoNEIT0=
> -----END CERTIFICATE-----
As you can see, the fingerprint comes from the certificate, but I got these errors messages in the fleet container:
> 'ca\_trusted\_fingerprint' set, looking for matching fingerprints"
Followed by
> no CA certificate matching the fingerprint
But now, if I change the output in the fleet settings to this:
Everything will work fine and I will get the logs inside the cluster, which I can confirm looking at the container logs:
> Connection to backoff(elasticsearch([https://elasticsearch:9200](https://elasticsearch:9200))) established
So, to resume, I have a CA certificate, the agent cannot send data if I use just the fingerprint of this certificate, but it can send data if I use the certificate in the advanced yaml configuration.
Of course I've could change my docker-compose to set the entire certificate in the `setup_fleet` step, but I'm trying to understand what is the issue here.
I've tested with both 8.13.4 and 8.14.1, same behaviour in both.
> [@stephenb](#):
>
> One quick look ... I don't see any [of these under fleet](https://www.elastic.co/guide/en/fleet/8.14/elastic-agent-cmd-options.html#elastic-agent-install-command)
> I would expect to see 1st or 2nd or 3 and 4
> I believe These are what are forwarded down to the agents (and used with the outputs)
No, these are used when enrolling the agent, I have no issues enrolling the agent that will work as the Fleet Server, I'm using the ca file in the enroll process.
Also, the default output created in Fleet does not use none of these settings, it creates a dummy `default` output pointing to `https://localhost:9200` that you have to manually change.
What I'm doing is using the Fleet API to change this during the setup process, and to make it easier I'm trying to use the fingerprint of the CA.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [June 29, 2024, 8:37pm UTC](https://discuss.elastic.co/t/elastic-agent-ca-trusted-fingerprint-does-not-working-but-the-certificate-works/362239/6 "2024-06-29T20:37:14Z")
Ok I am caught all the way up....
(I got hung up along the way on the `kibana_user.exist` you have to manually clean that up)
I am in the same state as you ... let me poke... weird...
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [June 29, 2024, 9:05pm UTC](https://discuss.elastic.co/t/elastic-agent-ca-trusted-fingerprint-does-not-working-but-the-certificate-works/362239/7 "2024-06-29T21:05:14Z")
@leandrojmp Found something ... and you are more of a cert expert than I.
So setup with your exact compose...
Then I put the CA fingerprint in metricbeat and test the output
Same error BUT if I set this with my other script that uses all the normal gen certs with our tool it works... (not saying you have to, but there must be some difference)
So seems to me ... something is wrong with a cert chain or the fingerprint
(or something wrong with the way we validate etc)
Last line seems to indicate
` ERROR x509: “SIEM” certificate is not standards compliant`
If you look at the 2 last logs... same as you are getting with the fleet...
```auto
./metricbeat -e test output
...
elasticsearch: https://localhost:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: ::1, 127.0.0.1
dial up... OK
TLS...
security: server's certificate chain verification is enabled
{"log.level":"info","@timestamp":"2024-06-29T14:02:28.165-0700","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.trustRootCA","file.name":"tlscommon/tls_config.go","file.line":179},"message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-06-29T14:02:28.165-0700","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.trustRootCA","file.name":"tlscommon/tls_config.go","file.line":208},"message":"no CA certificate matching the fingerprint","service.name":"metricbeat","ecs.version":"1.6.0"}
handshake... ERROR x509: “SIEM” certificate is not standards compliant
hyperion:metricbeat-8.14.1-darwin-x86_64 sbrown$
```
Looks like the code [here](https://github.com/elastic/elastic-agent-libs/blob/3dbdeaecd06e18ab9661e7f8038ed67e6ab024ee/transport/tlscommon/tls_config.go#L177)
And here is what it looks like when it matches this is just an elasticsearch doing the default setup.
```auto
elasticsearch: https://localhost:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: ::1, 127.0.0.1
dial up... OK
TLS...
security: server's certificate chain verification is enabled
{"log.level":"info","@timestamp":"2024-06-29T14:12:16.403-0700","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.trustRootCA","file.name":"tlscommon/tls_config.go","file.line":179},"message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-29T14:12:16.403-0700","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.trustRootCA","file.name":"tlscommon/tls_config.go","file.line":199},"message":"CA certificate matching 'ca_trusted_fingerprint' found, adding it to 'certificate_authorities'","service.name":"metricbeat","ecs.version":"1.6.0"}
handshake... OK
TLS version: TLSv1.3
dial up... OK
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [June 29, 2024, 9:24pm UTC](https://discuss.elastic.co/t/elastic-agent-ca-trusted-fingerprint-does-not-working-but-the-certificate-works/362239/8 "2024-06-29T21:24:18Z")
> [@stephenb](#):
>
> ERROR x509: “SIEM” certificate is not standards compliant
Oh Thanks!
This doesn't show up in the elastic agent logs, but this gives some direction on what to troubleshoot.
Maybe the way I'm creating the CA and certificates with `openssl` is not the correct way, I will look into it.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [June 29, 2024, 9:27pm UTC](https://discuss.elastic.co/t/elastic-agent-ca-trusted-fingerprint-does-not-working-but-the-certificate-works/362239/9 "2024-06-29T21:27:35Z")
Hard to debug... argh!
So when I get stuck like this...
I use Elastic tar.gz and metricbeat
Generate the certs etc... turn logging way up... once you get that working go back to docker etc... just a thought might speed up the dev / test cycle.
One thing I notice you are generating a CA with a .pem and a .key ... and when I see the default way the CA has no key as far as I can tell...
But also makes me wonder in the siem.pem (the SSL cert) is missing the "fullchain"
I notice that my default SSL cert is significantly longer than yours.
UPDATE
Looked some more... Think you need your `siem.pem` to be full chain
ca + cert when I decode mine that works... that is what is there...
Which kinda makes sense ... because you are asking the fingerprint (which is of the CA) to match a cert coming across the wire...
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [June 30, 2024, 12:53am UTC](https://discuss.elastic.co/t/elastic-agent-ca-trusted-fingerprint-does-not-working-but-the-certificate-works/362239/10 "2024-06-30T00:53:04Z")
> [@stephenb](#):
>
> Looked some more... Think you need your `siem.pem` to be full chain
> ca + cert when I decode mine that works... that is what is there...
Yeah, this was the issue indeed.
I've added the following line in my container that setup the certs:
> cat ca-siem.pem siem.pem \> siem.chain.pem
Then Elasticsearch stopped working with a SSL error, like this:
> Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL KeyManagerFactory
Inverting the order in the chain certificate solved it, so I first need to add the server certificate and after that the ca certificate.
> cat siem.pem ca-siem.pem \> siem.chain.pem
And have this in the settings:
```auto
- xpack.security.http.ssl.key=certs/siem-key.pem
- xpack.security.http.ssl.certificate=certs/siem.chain.pem
```
After that everything worked as expected and I got this message in the agent logs:
> CA certificate matching 'ca\_trusted\_fingerprint' found, adding it to 'certificate\_authorities'
I think that I understood how this work now.
The Elasticsearch server will present the agent with the certificate, and then the agent will try to match the `ca_trusted_fingerprint` with one of the certificates, if it can match, it will then add it to the `ssl.certificate_authorities` setting, this also explains why it works when adding directly to the `ssl.certificate_authorities`.
So, for this to work, you need to have the full chain, or at least the server + the ca certificates.
In the end the main issue was a small gap of knowledge on how SSL works in this case 😅 🤦♂️
Thanks!