How to resolve "Http client did not trust this server’s certificate"

Elastic Stack Elasticsearch
1

i get this error, despite following documentation for configuring elasticsearch on windows for HTTPS. I did the following steps:

STEP 1. Run CMD.EXE as ADMIN, Navigate to Elastic /BIN folder

elasticsearch-certutil ca

STEP 2: Accept default filename and enter a password

STEP 3:

elasticsearch-certutil cert --ca elastic-stack-ca.p12

STEP 4:

Accept default filename and enter a password

STEP 5:

Copy "elastic-certificates.p12" to the default elastic /config folder

STEP 6:

Enter the following in elastic YAML file, located in the elastic /config folder

# configure https
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.type: PKCS12
xpack.security.http.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.keystore.password: mypasswordvalue1234

STEP 7:

Run elasticsearch.bat at the ADMIN Console

STEP 8:

See the error "Http client did not trust this server’s certificate", with dejection

How can this error be resolved? Please talk to me like I am a second-grader.

2

You have two options:

  1. Change the server to use a certificate chain that the client already trusts
  2. Change the client so that it trusts the server's generated certificate.

What you can't do is change something on the server in order to make the client trust the server's existing certificate.
That's fundamental to the security of TLS - if the client doesn't trust the server's certificate then the server can't make the client do so. The client is in charge of who they trust - the server either needs to become trustworthy, or you need to change the client.

Option 1: Change the server certificate.

Elasticsearch auto-configuration generates its own CA (Certificate Authority), and then generates a server certificate for each node (that is signed by that CA).
This means that no existing client knows about your custom CA, and they don't trust it.

It has to be this way because there is (often) nothing trustworthy about a random Elasticsearch node running somewhere on the internet. I can configure my Elasticsearch node so that claims to be www.supermegabank.com, and security auto-configuration will generate a certificate for that address. But no client should actually (by default) trust that my server really does belong to Super Mega Bank. And the server can't make the client trust it because that would defeat one of the purposes of certificates - to know that you're really talking to the right server.

However, there are ways you can get a trusted certificate for your server.

  • Using an automated certificate issuer like Let's Encrypt
  • Using a commercial certificate issuer
  • Using a corporate certificate issuer

Each of those require that you have a real domain name for your server (but in the last case if doesn't have to be a public domain name).

If you're inside a corporate environment, talk to your internal team about how to get a corporate issued certificate for your nodes.

Otherwise, if you have a domain name that you control, and can make changes to, then look at https://letsencrypt.org/ and read through their instructions.

Option 2: Change the client configuration

This depends on what the client is. Unfortunately there are no universal standards, so you might have to do different things for different clients.

Official Elasticsearch language clients support a "CA fingerprint". When you configured Elasticsearch, the fingerprint should have been printed to the console, but you can also generate it with tools like openssl

Most clients will have some way of configuring a trusted Certificate Authority. The PEM formatted certificate for the cluster's certificate authority will be in your Elasticsearch configuration directory, in a file named http_ca.crt.
You will need to consult with your client tools about how to configure them to use that.
For example, curl supports a --cacert option.

3

Thank you for the informative reply, it's very helpful. I was able to do something interesting: On my Staging server I used a corporate certificate issued server-specific CERT, and my logstash/elastic configuration worked with HTTPS and I successfully indexed, along with re-configuration of PDF/Word built-in text extractor pipeline. On Production server, I used all the same YAML settings as Staging, along with Prod-server specific CERT issued by corporate, and I still get the error described in my original post. The CERT was issued quite some time ago, but it has not expired. I am going to ask corporate to re-issue the CERT for production and see if that works. It is group policy that all the servers trust the corporate root CA cert, so I know that is not the problem.

4

Hi, my internal CA re-keyed my production server https certificate but I still get "http client did not trust this server's certificate". I imported the public certificate into the Server "Personal" Trustore, and reference the physical copy of both the public and private key in YAML file. I used OpenSSL to extract the private key for use without a password. The YAML file looks like the following:

# ======================== Elasticsearch Configuration =========================

#

# NOTE: Elasticsearch comes with reasonable defaults for most settings.

#       Before you set out to tweak and tune the configuration, make sure you

#       understand what are you trying to accomplish and the consequences.

#

 

# Please consult the documentation for further information on configuration options:

# https://www.elastic.co/guide/en/elasticsearch/reference/index.html

#

 

cluster.name: my-elasticsearch-cluster

discovery.type: single-node

node.name: node-1

 

path.data: /apps/edssearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: false

 

network.host: 0.0.0.0

http.host: 0.0.0.0

http.port: 9200

http.max_content_length: 1500mb      

logger.level: DEBUG

 

xpack.security.enabled: true

xpack.security.http.ssl.enabled: true

xpack.security.http.ssl.client_authentication: optional

xpack.security.http.ssl.verification_mode: certificate

 

xpack.security.http.ssl.certificate: xxxxxx-mncp.crt

xpack.security.http.ssl.key: "xxxxxx-mncp.key"

 

xpack.security.http.ssl.certificate_authorities: ["chain_only.pem"]

 

indices.queries.cache.size: 20%

indices.memory.index_buffer_size: 60%

Any help to resolve this error would be greatly appreciated.

5

Hi, if anybody is willing to help me with my problem, it would be greatly appreciated. I have corporate internal CA issued certificates installed on the production server, with the YAML setting posted in the previous reply, but still get the "http client did not trust this server's certificate"

6

Apologies... are you still in the state that the elasticsearch will not start?

Or that some other client can not connect.

If Elasticsearch has started then try this.. .

have you run curl from

curl -v --cacert /path/to/ca.crt -u elastic https://<thesameiporhostnameyouuseintheclient>:9200

Run that command
Show the command
Show the results
This should show the why

If elasticsearch is still having issues... you need to talk to the folks that generate your certs as Elasticsearch uses certificates in the normal way .. we have many customer that use self-signed certs, Corporate Certs or Certs from Trused CA such as letsencrypt. There is nothing really unual about how elastic handle certs... but certs can be tricky if you do not work with them often.. if that is the case for you please reach out to your internal cert people / team.

7

Thanks for the reply and help. I went through with internal PKI team and compared the certs from staging with production, and they could not find any differences, other than ofcourse the CN and SAN. The certs work fine on staging, but not on prod. Could this be related to just a corrupted server? (i.e. messed of registry, java installation...)... we are considering rebuilding the prod server

8

Sure could be many things...

Still confused ..
a) elasticearch will not come up?
b) client can not connect?

9

this is kind of crazy, but in a good way....I rebooted the production server, and also I think group policy pushed some OS updates/patches, and I no longer get the error! and using powershell equivelent of the curl commands you provided, I get a Status 200 : OK. this is amazing, thank you for the help!

1 Like