SSL certificate | Let's Encrypt or self-signed

[Danelicious]

Hi there! I am trying to enrypt communication on my elastic server. I followed several tutorials. However, I always fail to succeed, so I am seeking for advise here. Elasticsearch is running fine after a fresh installation. But when I try to use certificates, it just won't start anymore.

Conditions

• Ubuntu 18.04

• Elasticsearch 7.4.2

• Single Node Cluster

What I tried (self-signed certificate)

I tried to follow the offical documentation, as well as this blog post.

The blog post seems straight forward, but I don't fully understand this part:
bin/elasticsearch-certutil ca --ca-dn CN=your_domain_name
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 -name "CN=your_domain_name,OU=Consulting Team,DC=your_domain_name,DC=com"

In specific the "-name" command confuses me, since it is actually there to create the files name.

Let's say I am running the command ...
bin/elasticsearch-certutil ca --ca-dn CN=test.example.com

... followd by ...
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 -name "CN=test.example,OU=Consulting Team,DC=your_domain_name,DC=com"

... it will create a file called "CN=test.example,OU=Consulting Team,DC=your_domain_name,DC=com". I also tried putting my servers IP address instead of the domain name but it also doesn't work.

It would be very helpful if someone could explain how the steps and commands to create the elastic-certificates.p12 file look like. I would appreciate examples with "test.example.com" or an IP-address. I strongly believe that it is only about those two commands. The part of copying files and creating folders is fairly easy.

Let's encrypt

Since I already encrypted communication on my domain with Let's encrypt, I have pem-files in following folder: /usr/local/psa/var/modules/letsencrypt/etc/archive/test.example.com.

Can't I use those to secure Elasticsearch and Kibana? I copied them to my config-certs-folder and adjusted the elastic configuration file, but it didn't work. How would the configuration look like with those certificates?

Last but not least my configuration file of Elasticsearch (elasticsearch.yml):

network.host: 127.0.0.1
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

Thanks in advance for your time!

[MiTschMR]

Hi @Danelicious

The medium blog post has a mistake in it. The parameter you need is not --name, but --ca-dn. A quick look into the elasticsearch-certutil help:

./bin/elasticsearch-certutil cert --help

Option               Description
------               -----------
-E <KeyValuePair>    Configure a setting
--ca                 path to an existing ca key pair (in PKCS#12 format)
--ca-cert            path to an existing ca certificate
--ca-dn              distinguished name to use for the generated ca. defaults to CN=Elastic Certificate Tool Autogenerated CA
--ca-key             path to an existing ca private key
--ca-pass            password for an existing ca private key or the generated ca private key
--days <Integer>     number of days that the generated certificates are valid
--dns                comma separated DNS names
-h, --help           show help
--in                 file containing details of the instances in yaml format
--ip                 comma separated IP addresses
--keep-ca-key        retain the CA private key for future use
--keysize <Integer>  size in bits of RSA keys
--multiple           generate files for multiple instances
--name               name of the generated certificate
--out                path to the output file that should be produced
--pass               password for generated private keys
--pem                output certificates and keys in PEM format instead of PKCS#12
-s, --silent         show minimal output
-v, --verbose        show verbose output

I advise you to use the Elastic blog post for creating certificates with the instance.yml as it is (imo) easier.

Let me know if you need additional help.

[Danelicious]

Hi @MiTschMR, thanks for your reply. I thought that "--name" must be a mistake. However, I still feel like the blog post is more straight forward. I get very confused on the Elastic blog post. It is also using a two node setup. Well, than I will try it again I guess. One question ... It says:

dns: [ 'node1.elastic.test.com' ]

"node1" is the "node.name" configured in the yml-file, right? So, if my Elastic instance is hosted on test.example.com, I would write:

instances:
  - name: 'node1'
    dns: [ 'node1.test.example.com' ]

And the kibana-part would be as follows?

- name: 'my-kibana'
    dns: [ 'kibana.local' ]

Where 'my-kibana' is set in the kibana.yml file under "server.name: my-kibana"? And dns must be specified as well (like my-kibana.test.example.com) or is it simply kibana.local as in the tutorial? Apologies if the questions are silly, I just got too confused already and don't understand fully.

Last but not least: Any ideas how to integrate Let's encrypt?

Thanks again for your time!

[MiTschMR]

Hi @Danelicious

I understand completely if you get confused. Let's try to work it out.

"node1" does not have to be the node.name configured in the yml-file, but it is recommended. The "name" in instances.yml describes how the certificate will be named.

I think you get confused with the domain names. Imagine having a domain controller with the domain abcd.com. Then you have server1, server2 and server3 as part of your domain. You can ping the servers with their names, server1.abcd.com, server2.abcd.com and server3.abcd.com. The names in the "dns" section must match the the fully qualified domain name of the servers or of their aliases.

With Kibana, it is the same. "name" describes the name and "dns" the FQDN.

Example:

instances.yml:

instances:
  - name: es01
    dns:
      - es01.abcd.com
 
  - name: es02
    dns:
      - es02.abcd.com
 
  - name: es03
    dns:
      - es03.abcd.com
 
  - name: kibana
    dns:
      - kibana.abcd.com

When the certificates are generated, you have to configure the settings in your configuration files of Elasticsearch and Kibana.

I don't know how to integrate Let's encrypt, sorry.

Hope this helps.

[Danelicious]

Thank you for your patience. The community here is awesome! I do have to thank you for encouraging me to keep on trying with the Elastic blog post because after all, I now got Elasticsearch running over https!!! How about that?

However, I feel the Elastic blog post also consists of a few "mistakes". I had to make a few adustments. Also the $ES_PATH_CONF and $ES_HOME is kind of "messed up" in this tutorial. The "cp"-commands are not very accurate or maybe I am just too silly . I will now try to get Kibana working as well, test everything and afterwards post a "solution" that worked for me.

Again, thanks for the clarification on the instances.yml-file and your patience! Cheers!

[Danelicious]

I wanted to give feedback to Kibana, as well as some follow up problems about ES.

1. I can access my ES cluster over https now via bowser (of course with a warning, but no problem). But unfortunately, if I curl via terminal (e.g. to create an index), I will get following warning message:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: curl - SSL CA Certificates

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

2. After creating the certificates, I will have 3 folders in /tmp/cert_blog/certs: 1) ca 2) elastic 3) kibana. CA-folder should only contain a .crt file. I suppose the other two folders should contain a .key, .crt file and the file ca.crt. However, only the ES-folder contains of all 3 files. In my kibana folder the ca.crt file is missing. That is not usual, right? So, I made a mistake? Because if I copy the ca.crt file from the ca-folder or ES-folder to the kibana-config-folder, kibana will start for 1s but afterwards stops again.

So, I guess my "pleasant certificate journey" will have to continue ...

[MiTschMR]

Hi @Danelicious

When I create the certificates with elasticsearch-certutil (with my example above) then there are the following folders:

• ca
• es01
• es02
• es03
• kibana

In the ca folder is only the ca.crt. In all other folders are the .crt and .key file. You already have the ca.crt file, that's why it is not included in them. I used the following commands:

$ bin/elasticsearch-certutil cert --silent --pem --in config/certificates/resources/instances.yml --out config/certificates/resources/certs/bundle.zip
$ unzip config/certificates/resources/certs/bundle.zip -d config/certificates/resources/certs

The kibana SSL configuration settings then look like (I'm using docker, so it is not accurate to non-docker):

SERVER_SSL_ENABLED=true
SERVER_SSL_KEY=/usr/share/kibana/config/certificates/kibana/kibana.key
SERVER_SSL_CERTIFICATE=/usr/share/kibana/config/certificates/kibana/kibana.crt
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/usr/share/kibana/config/certificates/ca/ca.crt
ELASTICSEARCH_SSL_VERIFICATIONMODE=certificate

Hope this helps clarifying.

[Danelicious]

Hey @MiTschMR!

Thanks for replicating the case. It's very helpful to know that the ca.crt file should only be in the folder you mentioned.

Just minutes ago and after days of trying, I actually now have an ES and Kibana instance running over https and being pw-protected. In my case, I made a silly mistake for the kibana issue (wrong path ). So, I apologize for taking your time on that but it was very helpful as I mentioned before.

I think I will need a break and hopefully some day I can use a Let's encrypt certificate, too.

Thank you so much and have a nice day!

[MiTschMR]

Hi @Danelicious

Great to hear that it is now working. Please mark the reply that was most helpful or was the solution as the solution so that other users with the same issue find the solution faster.

Cheers