Fleet Server Private Certificates Error

Elastic Stack Elasticsearch
1

I have a 3 node Elasticsearch cluster with certificates configured like this:

xpack.security.http.ssl:
  enabled: true
  key: certs/es1fqdn.key
  certificate: certs/es1fqdn.cer
  verification_mode: certificate
  certificate authorities: certs/ca/ca.crt

xpack.security.transport.ssl:
  enabled: true
  key: certs/es1fqdn.key
  certificate: certs/es1fqdn.cer
  verification_mode: certificate
  certificate authorities: certs/ca/ca.crt

And a single Kibana node configured like this:

server.ssl.enabled: true
server.ssl.certificate: /certs/fleetfqdn.crt
server.ssl.key: /certs/fleetfqdn.key
elasticsearch.ssl.certificateAuthorities: [/certs/ca/ca.crt]

The certificates were generated using a CSR from Elasticsearch and signed with a Microsoft CA. The ca.crt is in the format of RootCA and IssuingCA in a single certificate (Root CA is offline).

The Elasticsearch nodes and Kibana node appear to work with no issues, https is enabled and certificates are correct in the browser and appear to be trusted through the chain. Cluster is Green and I can manage everything in the Kibana web interface.

Where I have the problem is trying to add a Fleet server into the cluster. I've followed the documentation and the command I'm using to install the Fleet server is:

./elastic-agent install --url=https://fleetfqdn:8220 --fleet-server-es=https://esnode1fqdn:9200 --fleet-server-service-token=<tokenfromKibanainterface> --fleet-server-policy=policyname --fleet-server-es-ca-trusted-fingerprint=fingerprintfrom<ca.crt>onESsnode1 --certificate-authorities=/etc/fleet/ca.crt --fleet-server-cert=/etc/fleet/fleetfqdn.crt --fleet-server-cert-key=/etc/fleet/fleetfqdn.key --fleet-server-port=8220

The agent installs but then won't start and I see an error on Elasticsearch node 1:

AbstractHttpServerTransport (from the Fleet server IP)
SSLHandshakeException: bad_certficate

I've tried literally everything to connect the Fleet server but nothing works. Can anyone help?

2

What's in this file?

My best guess is that it contains the leaf certificate & the intermediate, but not the root.

Fingerprint based trust only works if the server provides a copy of the CA. The client (fleet) uses the fingerprint to check that the provided CA has the expected fingerprint. It cannot do that if the CA isn't in the server's chain.

If you want the ES node to provide the CA in the chain, then you need to append the CA to certs/es1fqdn.cer

3

It originally contained just the certificate for the node 1 server. I have also tried appending the leaf certificate.

So leaf and root need to be appended? Which fingerprint should i use then? Fingerprint of the leafe or the root? Its only Fleet that seems to have an issue with the current setup.

4

I believe Fleet requires a fingerprint for the root certificate, and the root certificate needs to be in the server's chain.

CA fingerprinting is primarily designed to work with Elasticsearch's security auto-configuration which does not generate an intermediate, and does include the CA in the chain.
It can be used for other configurations, but its goal is to simplify out-of-the-box setups rather than being a universal solution.

Did you configure other products to use the fingerprint or directly reference the CA?

5

@TimV

They are configured exactly as I've put the config files above. I reference the ca.crt file which has both the leaf and root in

6

@TimV
I've just tried appending the leaf and root into the Elastic certificates. When installing the Fleet server I get the error "Error - failed version compatibility check with elasticsearch: x509: certificate signed by an unknown authority"

Elasticsearch reports "bad _certificate" again.

7

I've got it working, so it looks like you need the leaf and root in the Elastic host certificates and then run the Fleet installer without the fingerprint but use ----fleet-server-es-ca and point it at the ca.crt file on the Fleet server.

I wish the documentation would cover things outside the default setup, I'd have thought this was quite a common install method - there's certainly a lot of unresolved forum posts about TLS/SSL.

8

@TimV
I was wrong, it's still broken. Same error in the Elasticsearch hosts log. Only found it's still a problem when configuring Elastic Defend and the integration fails on the endpoint as it can't talk to the Elasticsearch servers.

I now have the chain in the certificate on all Elasticsearch hosts and used the fleet-server-es-ca option to point at the ca.crt which contains the leaf and root certificate.

There really needs to be some documentation for self-signed certificates, I can't find any documentation anywhere.