HTTP Certificates when CA is a chain

I'm following the documentation here: Tutorial 2: Securing a self-managed Elastic Stack | Elastic Installation and Upgrade Guide [8.14] | Elastic and have got to Step 3.

The CA I've exported from our Microsoft Enterprise CA but it's a Two Tier authority with a Root and Subordinate. This means the CA certificate is a chain (containing the Root and Sub certificates).

When I try to create the http certificates I can't use this CA certificate as it's a chain. What do you do if your CA is a chain? I can't see any documentation on this?

Thanks

What's the exact problem you're running into?

The elastic certutil tool tells me I can use a certificate chain and it must be a single certificate.

I'm sorry, you need to give more detail or we're just guessing about what problem you're running into.

What are you trying to use certutil for? How are you invoking it? What is the error message.

I am surprised that you're trying to use certutil at all - if you're using Microsoft certificate tooling then I would expect that you'd just stick with that and generate certificates there.

I'm using certutil on elastic to generate the csr which I then sign with the MS CA. I'm following the documentation I linked at step 3. I can't generate the http certificates as elastic tells me it doesn't support certificate chains as the ca.crt.

Sorry, you're still not providing me with the detail that would allow me to understand where this is failing.

Can you copy-and-paste the exact commands and output?

I've exported my Subordinate Root CA cert and key from my Windows subordinate CA. I've then placed these files in /etc/elasticsearch/certs/ca. I've then run the command to import the ca into the keystore.

sudo /usr/share/elasticsearch/jdk/bin/keytool -importcert -trustcacerts -noprompt -keystore /etc/elasticsearch/certs/elastic-stack-ca.p12 -storepass -alias new-ca -file /etc/elasticsearch/certs/ca/ca.crt

This appears to work ok. I've then generated certificates for the transport layer following Step 2 of the documentation posted earlier. This uses certutil to generate the csrs (including any SANS/IPs) which I then pass to the Windows CA for signing. The resulting certificates are then stored in /etc/elasticsearch/certs and referenced in the elasticsearch.yml.

xpack.security.transport.ssl:
keystore.path: /etc/elasticsearch/certs/fqdnofhost.cer
truststore.path: /etc/elasticsearch/certs/ca/ca.crt

The issue comes when I get to Step 3 - Generating http certificates.

sudo /usr/share/elasticsearch/bin/elasticsearch-certutil http

I then follow the documentation:

Respond to the command prompts as follows:
*** When asked if you want to generate a CSR, enter n.**
*** When asked if you want to use an existing CA, enter y.**
If you’re using your organization’s CA certificate, specify that certificate and key in the following two steps.

The elasticsearch http tool then asks me to specify the location of the existing ca file. I enter /etc/elasticsearch/certs/ca/ca.crt

I receive the error:

The file at /etc/elasticsearch/certs/ca/ca.crt contains multiple certificates. This type of file typically represents a certificate-chain
This tool requires a single certificate for the CA
Error: /etc/elasticsearch/certs/ca/ca.crt: Unsupported file type (certificate chain), with exit code 65

As I mentioned earlier, if you have existing certificate tooling (your Microsoft suite) then that is usually the best option. There is absolutely no requirement to use elasticsearch-certutil, and if you do chose to use it, then you may prefer to generate a CSR that you can fulfill in your MS tooling instead of exporting the CA from the Microsoft CA manager.
That recommendation is from both a convenience point of view (stick with one set of tools that you know well) and a security point of view (exporting CA keys and using them in multiple places is likely to lead to them being accidentally leaked).

If you really want to use elasticsearch-certutil to generate your certificates then you're going to need to edit the ca.crt file so that it contains only a single certificate.

Your ca.crt probably looks something like the example at the bottom of this post.
It contains two certificates separated by -----BEGIN and ----END markers. Typically the first certificate will be your intermediate (subordinate/signing) cert, and the second certificate will be the root.
You just need to copy the first certificate into a separate file and point elasticsearch-certutil at that file instead.

-----BEGIN CERTIFICATE-----
MIIEkDCCAnigAwIBAgIVAKZj17ubGk9QgBSS5NnSoy7TALnMMA0GCSqGSIb3DQEB
CwUAMFMxEzARBgoJkiaJk/IsZAEZFgNuZXQxFzAVBgoJkiaJk/IsZAEZFgdleGFt
cGxlMREwDwYDVQQLEwhzZWN1cml0eTEQMA4GA1UEAxMHUm9vdCBDQTAeFw0yMjA0
MDYwODA5MTRaFw00NjExMjYwODA5MTRaMFsxEzARBgoJkiaJk/IsZAEZFgNuZXQx
FzAVBgoJkiaJk/IsZAEZFgdleGFtcGxlMREwDwYDVQQLEwhzZWN1cml0eTEYMBYG
A1UEAxMPSW50ZXJtZWRpYXRlIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAybkCUxBXL853y5sdH9Lgy1+bk6/z3bsJ0i1Qn6Vy2Rr/QNmExVxZ3GB0
J4GKcnM2zgLc0fGpiVfAxBUZQB3RRekFelK5+y83q54QkLnuSF1llovlbH9cJaeo
DD7mllKHsqK3QfNw1kAu6xTkLeegkTHZrUyRUw6Pgo0RsaIHKYorlKE2zVKsvCrI
2skADG9aSpb9k1nAndNoKQD1vjoSkhlQeoJtliUAqtFsHw0SOJzmYvTZ9PJVDOsg
hAYnx+v3LsP4SejYp1B//o9X1bbF2HZbL6RyboGLnUi5vX/aVD8tZoFgZQkJ67tz
Ml3pOGygYwmCrISD60xiIB5cutL13wIDAQABo1MwUTAdBgNVHQ4EFgQUcpEkgMyQ
ekd3W5UZA13qCzFSpRAwHwYDVR0jBBgwFoAUeInVDXfJZYa7rAXDkS+BfFVT5IMw
DwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAgEAu74tFsGJS74ElxA/
QYwyKomrhNGyfdcWt49QOrBjkortz3kNrAvJKKnZEuLTx4KzFOYaq5+Xuwr+3C5Z
EEF5uTLa+ENI0su37PPmYbDeCkO1ugwntbqpoZGJvOKkegAWwnCX0AV2QIKpLGYj
5WV6MNOCt1udchzOFFUBfGpMz1YJUi3to3FaJAoy6pbUBlj1hKEhJ5ans86j9gCZ
HM8MatfN/N7JFfAkyeLVXXe09eaqwNFa23fV8RJCwAXuywVGjOmepCs8J7uBkznE
I5hbl0D4u250SM796cTOnp3zo2eSs2g6gMw79aweFvwxjfzF/JMPB6eTSZz54AI7
xcw+8HsE5ZtWF440RWmjQiD82zD4Um5K2r5HcT7CBYOncLGklsZ9WKn7W7bR+Yw2
HAd6TzU8q1MgLRzPCCtnsQDPNVrrhyt+jXbXkPtZk7V6fmV6N2NVfkzT3npVa3vU
DlIR8NqZO/zZ8NMerdZDlLc0caSI2vTEuqx1YI3I2l0jsDkAv+Y4B6CSMrl0rD+u
QXo3ve5Q+B7zsdNw27xS2Vt2mkxPF4GRF8Py0nKy2GR2L+n5VusDi6ulDeLJaXMm
Kw2GDSYMTNEaBBhkk10CFastCP8QE1U1rByKGSbkla7wnZr4JfkadviQQ2K4JE1N
oaUHAOcaUsYJbT5zqqjYMWutSkQ=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFhzCCA2+gAwIBAgIUIuKKXb4R3VSsrNtfisg4tAEiE3UwDQYJKoZIhvcNAQEL
BQAwUzETMBEGCgmSJomT8ixkARkWA25ldDEXMBUGCgmSJomT8ixkARkWB2V4YW1w
bGUxETAPBgNVBAsTCHNlY3VyaXR5MRAwDgYDVQQDEwdSb290IENBMB4XDTIyMDQw
NjAzMzUzNFoXDTQ5MDgyMTAzMzUzNFowUzETMBEGCgmSJomT8ixkARkWA25ldDEX
MBUGCgmSJomT8ixkARkWB2V4YW1wbGUxETAPBgNVBAsTCHNlY3VyaXR5MRAwDgYD
VQQDEwdSb290IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvJHJ
75bH97QPG6lFsLbOUrb9v6ItL2yVse1zdkDOff0k4/pmxn5B+5t7faTlWuyuXRuH
t1wFvGjo+xqDKNyJ5NOcntVP78NZwT4eeRBJfSQFU5HlxLWJkvFRbD/J1G7wVxgI
90Pt34Ql4djR03e/F7O6c8Za0xCAJqC3d6Ah4OTUDORyS51DF21yxkJtpi2xxWin
TeM6xu7JUBjJy1XKEEzsZsgn0o4/T9irdaErbJPJDp1tGjmaH4XP3FuIn4jNIiwf
z75VjauSWeO6tYc5gG4GDS2NiVpMDliH1mqWest1R1TS6+FLBW1mTRU5dxuXCbI7
5oY41KqLj+5xjD2V9LnsWpSjIhuw2bbC5F/ILi6rCFdWFGfCnygG1H29O/RWibST
s6VbcShRqk9HsP2a/HE3V93U9Cyv7tIBM5L2sowOU8qt7b6N5BiL/3DQsLGSeXhl
4udwuqTd4jWWnvXL5GzOnr0HQI/bEzEC0/LjxGqzmdPtrAx0xIeu4sdSzz39nxz5
yhWGacwNI0mCSH1BRYf74QciDh+Pg9QDWqUNo4we+36WamXDgY4ZcO91Ltwh3JII
oQZ6T310fkwRx0U9TCj3zKAKnPw/HoRRQUAZBeTR7Ndc3fRf6PRlgMlmRP3L1ko2
0C4RngMHtOi7cmqF9vxtjM1EB236REtkcWSPzbECAwEAAaNTMFEwHQYDVR0OBBYE
FHiJ1Q13yWWGu6wFw5EvgXxVU+SDMB8GA1UdIwQYMBaAFHiJ1Q13yWWGu6wFw5Ev
gXxVU+SDMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggIBAIrnaGWJ
1hCRaUJde5MfkJTFEjnA4CxH8EoIYIDGkO94/NuHXyKuCKO1KouIlxcOVqtGFHyU
t0gJvrgY+Pw/V5xg9w+HDA0jtzKvq36sLFPPitdeVHiOsWv0BjypC3VU395RFMOq
jDdjLy48hdq8j3i840DCqq9py/tpabASiG57Eg8c2Y+9GL5neWfctnS+HABuRoKy
y7kCThuRxS7Wnee92A6jPC1jyXC7rIjmL4IvhMCVYaCeO0b6TSONFUFRZCmnvwSx
YvakYNaO9CFTkxcLIvX/EDpO0ARA+9I1Yetev1tfBHJIL1iyNoRF8VQPikuBBeT6
oBDvLyE70hk0W4h5fIO3QKDx7q+5Ss7R7CNYXcvaXLwDven3cck5pjjpEJ3/XSv2
EYcwY8baQIoqIL0/myGnLeJPQZsR8lB+M/pRmjmkJDASFjfzWkChTeUG/tFnw4nu
s5NDEq0A5Q1tJfKDYltP+BJPGrptCjMMZji/uDXWMIOQok3M/keEmTevYxGdW+nQ
iccdq8CMv4K4sJFhQA2oLKttznCh02hDJJ8abXNYh0RITBH0r2aKHSP7c633vAfw
kGk77CtkdxhDfsaqycJACvbZLk9br96zQIp7/cUowXbqLrASBN7Tz7jl6Wry5DbH
hlBIj7OY7pwNJggD2jBwc42QxTWDH/SNixIo
-----END CERTIFICATE-----

I've done exactly as you've suggested and elasticsearch doesn't start with a bad_certificate error.

Is there any documentation anywhere as to how to configure elasticsearch using a Microsoft (or other) CA?

I'm assuming I can just use the certutil tool to generate the csrs for each node, kibana and fleet and then just stick the certs as pem format somewhere and then reference in the .yml. I can ignore all the keystore stuff at that point.

I've managed to get Elasticsearch running by just using PEMs. However it doesn't seem to be documented anywhere that you can't use certificate verification without having a specific certificate template created by your CA. The MS webserver template doesn't include client authentication by default. It would be good if the certificate requirements were documented somewhere (did I miss it?).

Also in Elasticsearch.yml you can specify the /etc/elaticsearch/certs directory with just /certs/. In Kibana you appear to have to put in the full path, /certs/ doesn't seem to work. Is that right?

I'm now trying to secure the Fleet server but it appears you have to start again with Fleet as there is no documented method to update the installation to use certificates?

Fleet seems to have very limited documentation on using CA signed certificates. I can't get the Fleet/Elastic Agent connected to Kibana.

sudo ./elastic-agent install --url=https://fleetserverfqdn:443
--fleet-server-es=https://elasticserverfqdn:9200
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjA4MTQyODAyOTg6NXdFYjdzcUtRMUtKRjRRUndpMlRJUQ
--fleet-server-policy=fleet-server-policy
--certificate-authorities=/etc/fleet/ca.crt
--fleet-server-cert=/etc/fleet/fleetserver.crt
--fleet-server-cert-key=/etc/fleet/fleetserver.key
--fleet-server-port=8220

results in Error - x509: certificate signed by unknown authority.

I've tried adding --fleet-server-es-ca-trusted-fingerprint with the fingerprint of the ca.crt I'm using on each ES server and Kibana but same result.

What am I doing wrong here?

What's in ca.crt is it the chain or just the root cert?

What I would normally do in your case is:

  1. configure ES to send the whole chain (or at least the leaf & intermediate)
  2. configure clients to trust the root only.

Configuring ES to use the chain can be done by concatenating the certificates together and using that chained certificate file for xpack.security.http.ssl.certificate
You can follow the steps here

The ca.crt has both the Root and Subordinate certificate in.

I assume I don't need to add the chain to the ES certs if the ca.crt already includes both the root and sub?

Certs are currently working for ES and Kibana, I can't turn on Full or Certificate validation though without getting errors about not being able to verify the certs.

Just having issues connecting Fleet (results in Error - x509: certificate signed by unknown authority). This error is seen on the fleet server and on my primary elasticsearch server.