HTTPS config for Elasticsearch

Elastic Stack Elasticsearch
,
1

I've been stuck for a few days tying to enable HTTPS for public access from my elasticsearch javascript client. My elasticsearch server is running on the host machine.

The server is only reachable using HTTP:
curl -k -u "elastic:password" http://193.100.00.00:9200

The problem initially was that I don't have the password to the /etc/elasticsearch/certs/http_ca.crt file

So I can't use it in my elasticsearch.yml file when enabling ssl configuration.

xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.password: <your-keystore-password>

Here is 1 solution I did but it's NOT CORRECT because I get "detected self-signed certificate in chain" error when trying to connect from the client:

To manually enable HTTPS, I followed the following steps:

  • Generate a self-signed CA: sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca

  • MV file to certs directory: mv /usr/share/elasticsearch/elastic-stack-ca.p12 /etc/elasticsearch/certs/

  • Generate SSL/TCL certs: sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-stack-ca.p12

  • Set file permissions:

    sudo chown elasticsearch:elasticsearch /etc/elasticsearch/certs/elastic-certificates.p12
sudo chmod 600 /etc/elasticsearch/certs/elastic-certificates.p12

  • Add the following config the elasticsearch.yml file:

    xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.password: <your-keystore-password>
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.password: <your-truststore-password>

With this config, the command will work correctly:
curl -k -u "elastic:password" https://193.100.00.00:9200

But from client, I get: "detected self-signed certificate in chain"

I've tried generating a certificate from a trusted CA (let's encrypt), but this solution doesn't make sense to me because I can't maintain auto-renewal through certbot cause I have to:

  • Convert to p12 format for the elasticsearch.yml file
  • Copy the CA to my client parameter

I'm not good with this stuff, so I'm sure i'm over complicating it. But if someone can really just tell me what to do I would greatly appreciate it. There's too many documents with bits and pieces of information and I feel like there's probably a straight-forward answer to enable HTTPS on an elastic-search server to connect with the elasticsearch javascript client using ssl.

Any help would be great. Thanks!

2

I do not use javascript, but how are you creating the client in your code?

You need to specify the CA you used to create the HTTP certificate or tell the client to ignore validation and accept self-signed CA.

There is an example of this in the Connection part of the Elasticsearch JavaScript documentation.

If I'm not wrong you would need to add something like this while creating your client:

  tls: {
    rejectUnauthorized: false
  }
3

Hi,

thanks for the additional info.

I've instantiated my client very simply with:

auth: {
    username: process.env.ELASTICSEARCH_USERNAME ?? 'elastic',
    password: process.env.ELASTICSEARCH_PASSWORD ?? 'elastic',
  },
  tls: {
    ca: fs.readFileSync('http_ca.crt'),
  },

Unfortunately, for production I can't set rejectUnauthorized: false as that would be less secure than just keeping my HTTP CA from my understanding

4

Passing the CA file should work if the CA file is correct and was the same used to create the certificate.

Is the http_ca.crt the same CA that you generated with this command /usr/share/elasticsearch/bin/elasticsearch-certutil ca ?

Normaly, http_ca.crt is the automatically generate CA for the http endpoint of elasticsearch, since you said in the previous post that you generated another self-signed CA, this file may not be the one that should be used in your configuration.

What do you have when you use this file in curl?

curl -vvv -u "elastic:password" https://193.100.00.00:9200 --cacert same-file-used-in-your-code
5

The curl command will work correctly when I generate my own self-signed CA file and use that.

The issue is when I plug this new CA file into my client instantiation, I get the Error "detected self-signed certificate in chain". I think the client doesn't trust the connection because I generated my own CA certificate on the server

6

Can you share the result of the curl command mentioned in the previous post?

It is the one where you use --cacert, you cannot use -k as this will ignore validation.

Please run the command:

curl -vvv -u "elastic:password" https://193.100.00.00:9200 --cacert your-ca-file

And share the entire result.

Yeah, but you are telling your client to use the CA, so it should work unless something is wrong with the CA or with your configuration.

7

I see. Ok, I'll try right now. One moment

8

Here is the output of the command you shared:

Desktop $curl -vvv -u "elastic:" https://xx:9200 --cacert ./ca-cert.crt 
*   Trying xx:9200...
* Connected to xx (xx) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: ./ca-cert.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=instance
*  start date: Aug 16 22:52:00 2024 GMT
*  expire date: Aug 16 22:52:00 2027 GMT
* SSL: certificate subject name 'instance' does not match target host name 'xx'
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (60) SSL: certificate subject name 'instance' does not match target host name 'xx'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

xx is the IP address of the server that I blocked out.

It's also worth mentioning that the output of elasticsearch-certutil ca is a p12 format file. but the CA parameter on the client expects a PEM format file so I used the following command to the encoded PEM certificate from the p12 file and use that:
openssl pkcs12 -in /path/to/elastic-stack-ca.p12 -clcerts -nokeys -out ca-cert.crt

9

Perhaps look at

You need to create a cert with the proper SANs and / or IPs

10

The issue seems to be your certificate, the curl command also didn't worked as you can see by the following message.

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it.

Your certificate is not valid for the host you made the request:

You need to create a certificate for the http endpoint with the correct hostname.

This documentation may help.

1 Like
11

After reviewing some of the documentation, I changed to use sudo /usr/share/elasticsearch/bin/elasticsearch-certutil http for certificate generation.

After using the new certificates, here is the output of the curl command:

Desktop $curl -vvv -u "elastic:" https://xx:9200 --cacert ./http-ca-cert.crt 
*   Trying xx:9200...
* Connected to xx (xx) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: ./http-ca-cert.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: DC=io; DC=domain; CN=subdomain
*  start date: Aug 18 05:14:23 2024 GMT
*  expire date: Aug 18 05:14:23 2029 GMT
*  subjectAltName: host "xx" matched cert's IP address!
*  issuer: CN=xx
*  SSL certificate verify ok.
* Server auth using Basic with user 'elastic'
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: xx:9200
> Authorization: Basic 
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 548
< 
{
  "name" : "elk-stack-elasticsearch",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "",
  "version" : {
    "number" : "8.15.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "",
    "build_date" : "2024-08-05T10:05:34.233336849Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host xx left intact

To me this looks like it's working now but when plugging the generated CA into my client, I still get:
Ping to Elasticsearch failed with error "self-signed certificate in certificate chain"

12

I saw the recommendation to use certificate_authorities instead of using the other two parameters but I haven't had success setting this config. The elasticsearch.service has an error when trying to restart the service.

The only other recommendation I found is to "generate certs with proper SANs, IPs" which I believe they're correct and don't have the experience to think I set them up incorrectly

13

Yes the curl is working and the cert matches.

What client exactly and can you share the entire connection code snippet?

14

It seems to be working now. The issue was that I was trying to create a crt file from my Dockerfile to load it into my production server:

# Dockerfile
ENV ELASTICSEARCH_CA_CERT=${ELASTICSEARCH_CA_CERT}
#Create crt file for elasticsearch
RUN echo ${ELASTICSEARCH_CA_CERT} > ./http_ca.crt

But seems to have an issue. I switched to simply just assign the certificate directly to the environment variable as a string and using that in my client instantiation which seems to have fixed the issue.

The only thing I see missing from my workflow for a perfect elasticsearch implementation would be auto-renewal of the certificate. but manual update every couple years is not terrible so it can work.

Thank you @stephenb and @leandrojmp for the additional troubleshooting help!