Configure TLS production environment with own CA

Hi everyone,
I'm having a bit of an hard time configuring TLS in a production environment. I already have TLS running on a test environment (with autoconfiguration, using .deb package). At the end of the procedure it prints something like this:
For production cluster deployments you will need to replace ca and HTTP and transport certificates with those from your enterprise CA

So now I'd like to ask:

1- if this will improve security in any meaningful way or (from what I've read) it's just a convenience since the CA would be already present of servers.

2- if it's needed, how can it be configured.

This is what a tried so far (I have a wildcard certificate from a CA):

  • configuring pem certificate + key + ca but it complains about a missing keystore when I try to generate an enrollment token:
./bin/elasticsearch-create-enrollment-token -s node --url "https://myfqdn:9200"
Unable to create enrollment token for scope [node]

ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration is not configured with a keystore
  • creating a keystore with openssl pkcs12 -export -out keystore.p12 -inkey cert.key -in cert.crt without password but it fails because of a missing password.
  • creating the above keystore with a password, added the password with:
./elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./elasticsearch-keystore add xpack.security.transport.ssl.trustore.secure_password
./elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

the keystore is created from a file with chained certificate in the correct order. I tried to create one with server certificate + ca certificate + intermediate + root ca (and certificate + ca and certificate only), with
openssl pkcs12 -export -out keystore.p12 -inkey mykey.key -in mycert.crt

if I inspect the keystore:

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: 1
Creation date: Jul 28, 2022
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
...

I can see 3 certificates in there but elasticsearch doesn't start with the error below:
Caused by: org.elasticsearch.common.ssl.SslConfigException: the truststore [/etc/elasticsearch/certs/xxx.p12] does not contain any trusted certificate entries.

This is (part of) my elasticserch.yml just in case:

xpack.security.enabled: true

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/keystore.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/keystore.p12
  truststore.path: certs/keystore.p12

Thanks for your help.

Can you point me to where you read that? The auto configuration process is entirely supported for production use cases, and if there's something in the install process (or docs?) that implies otherwise, we'd like to fix it.

Many people might prefer to replace the HTTP certificate with one that is issued by an existing CA - that way your clients are more likely to trust the certificate out-of-the-box, but that's really more about convenience/ease-of-use than it is about security.

if this will improve security in any meaningful way or (from what I've read) it's just a convenience since the CA would be already present of servers.

It is primarily about convenience.
There is 1 area where security might come in to it. In order to support the enrollment of additional nodes (which might have different IPs/hostnames), the ES nodes keep a copy of the CA key in the HTTP PKCS#12 keystore so that it can generate new certificates on the fly.
Depending on how widely you are trusting that CA, you might not want the key to be kept in an online system. In that case you can either remove the CA key from the system (which will prevent your from using the streamlined enrollment process) or replace the CA entirely.

but it complains about a missing keystore when I try to generate an enrollment token

If you opt not to use the auto-generated HTTP CA, then you cannot use the token-based node enrollment anymore. It only works with the auto-generated setup.
That is, if you decide to configure TLS yourself, then you need to do it all yourself, the enrollment process isn't available anymore.
There's no real magic in the enrollment process - it's just a way to replicate the auto generated setup from one seed node to additional nodes. It's fairly simple to do that yourself, although I suspect the docs are a little weak in that area.

1 Like

Thanks a lot for your clarifications.

Can you point me to where you read that? The auto configuration process is entirely supported for production use cases, and if there's something in the install process (or docs?) that implies otherwise, we'd like to fix it.

Unfortunately I thought it was some official documentation but it wasn't. I found it here Elasticsearch 8.x Deployment. What changed? | by vikas yadav | DevOps Dudes | Medium
I already commented that article pointing out that that information is not correct.

Thanks for clarifying the security aspect, since everything is on a private network and the use of the CA is limited to few cluster nodes and clients I will use the automatic setup.