Configure TLS production environment with own CA

Elastic Stack Elasticsearch
1

Hi everyone,
I'm having a bit of an hard time configuring TLS in a production environment. I already have TLS running on a test environment (with autoconfiguration, using .deb package). At the end of the procedure it prints something like this:
For production cluster deployments you will need to replace ca and HTTP and transport certificates with those from your enterprise CA

So now I'd like to ask:

1- if this will improve security in any meaningful way or (from what I've read) it's just a convenience since the CA would be already present of servers.

2- if it's needed, how can it be configured.

This is what a tried so far (I have a wildcard certificate from a CA):

  • configuring pem certificate + key + ca but it complains about a missing keystore when I try to generate an enrollment token:
./bin/elasticsearch-create-enrollment-token -s node --url "https://myfqdn:9200"
Unable to create enrollment token for scope [node]

ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration is not configured with a keystore
  • creating a keystore with openssl pkcs12 -export -out keystore.p12 -inkey cert.key -in cert.crt without password but it fails because of a missing password.
  • creating the above keystore with a password, added the password with:
./elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./elasticsearch-keystore add xpack.security.transport.ssl.trustore.secure_password
./elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

the keystore is created from a file with chained certificate in the correct order. I tried to create one with server certificate + ca certificate + intermediate + root ca (and certificate + ca and certificate only), with
openssl pkcs12 -export -out keystore.p12 -inkey mykey.key -in mycert.crt

if I inspect the keystore:

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: 1
Creation date: Jul 28, 2022
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
...

I can see 3 certificates in there but elasticsearch doesn't start with the error below:
Caused by: org.elasticsearch.common.ssl.SslConfigException: the truststore [/etc/elasticsearch/certs/xxx.p12] does not contain any trusted certificate entries.

This is (part of) my elasticserch.yml just in case:

xpack.security.enabled: true

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/keystore.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/keystore.p12
  truststore.path: certs/keystore.p12

Thanks for your help.

3

Can you point me to where you read that? The auto configuration process is entirely supported for production use cases, and if there's something in the install process (or docs?) that implies otherwise, we'd like to fix it.

Many people might prefer to replace the HTTP certificate with one that is issued by an existing CA - that way your clients are more likely to trust the certificate out-of-the-box, but that's really more about convenience/ease-of-use than it is about security.

if this will improve security in any meaningful way or (from what I've read) it's just a convenience since the CA would be already present of servers.

It is primarily about convenience.
There is 1 area where security might come in to it. In order to support the enrollment of additional nodes (which might have different IPs/hostnames), the ES nodes keep a copy of the CA key in the HTTP PKCS#12 keystore so that it can generate new certificates on the fly.
Depending on how widely you are trusting that CA, you might not want the key to be kept in an online system. In that case you can either remove the CA key from the system (which will prevent your from using the streamlined enrollment process) or replace the CA entirely.

but it complains about a missing keystore when I try to generate an enrollment token

If you opt not to use the auto-generated HTTP CA, then you cannot use the token-based node enrollment anymore. It only works with the auto-generated setup.
That is, if you decide to configure TLS yourself, then you need to do it all yourself, the enrollment process isn't available anymore.
There's no real magic in the enrollment process - it's just a way to replicate the auto generated setup from one seed node to additional nodes. It's fairly simple to do that yourself, although I suspect the docs are a little weak in that area.

1 Like
4

Thanks a lot for your clarifications.

Can you point me to where you read that? The auto configuration process is entirely supported for production use cases, and if there's something in the install process (or docs?) that implies otherwise, we'd like to fix it.

Unfortunately I thought it was some official documentation but it wasn't. I found it here Elasticsearch 8.x Deployment. What changed? | by vikas yadav | DevOps Dudes | Medium
I already commented that article pointing out that that information is not correct.

Thanks for clarifying the security aspect, since everything is on a private network and the use of the CA is limited to few cluster nodes and clients I will use the automatic setup.

5

Unfortunately I have to correct myself: this is where I read about the autogenerated CA not being recommended for production. I guess you already had outlined why. Am I correct?

What happens if you enroll Fleet Server without specifying certificates?

If the certificates are managed by your organization and installed at the system level, they will be used to encrypt traffic between Elastic Agents, Fleet Server, and Elasticsearch.

If system-level certificates don’t exist, Fleet Server automatically generates self-signed certificates. Traffic between Fleet Server and Elastic Agents over HTTPS is encrypted, but the certificate chain cannot be verified. Any Elastic Agents enrolling in Fleet Server will need to pass the --insecure flag to acknowledge that the certificate chain is not verified.

Allowing Fleet Server to generate self-signed certificates is useful to get things running for development, but not recommended in a production environment.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.