How to use elasticsearch-setup-passwords with TLS and selfsigned certificates?

Elastic Stack Elasticsearch
1

Hi,

I ran an existing Cluster 7.2.0 without xpack security in kubernetes. I am using self signed pem certificates which I have created with elasticsearch-certutils.

Configuration is completely done via environment variables.

After enabling Security and adding TLS layer I want to create the initial passwords using elasticsearch-setup-passwords script. Here is the output: first my environment filtered for xpack, then the real command:

kubectl exec -it poc-es-master-0 -- /bin/bash -e -c 'env | grep xpack && bin/elasticsearch-setup-passwords auto --url https://poc-es-http:9200'
xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/http/ca.crt
xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/transport/ca.crt
xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/http/tls.crt
xpack.security.transport.ssl.client_authentication=optional
xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/http/tls.key
xpack.security.transport.ssl.enabled=true
xpack.security.http.ssl.client_authentication=optional
xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/transport/tls.key
xpack.security.transport.ssl.verification_mode=certificate
xpack.security.http.ssl.enabled=true
xpack.security.enabled=true
xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/transport/tls.crt

SSL connection to https://localhost:9200/_security/_authenticate?pretty failed: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Please check the elasticsearch SSL settings under xpack.security.http.ssl.

ERROR: Failed to establish SSL connection to elasticsearch at https://localhost:9200/_security/_authenticate?pretty.

So as I see it, the environment variables are there, but they are not loaded by elasticsearch-setup-passwords script.

How can I tell the script either NOT to check certificates or better to check against the given CA?

poc-es-http is the service which routes to http rest api of elasticsearch. The certificate is issued to this CN.

Thanks,
Andreas

2

Is the tls.crt certificate signed by the ca.crt CA certificate ? Because the setup passwords tool is configured to trust ca.crt

xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/http/ca.crt

and your Elasticsearch nodes are using

xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/http/tls.crt
xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/http/tls.key

for TLS on the http layer.

xpack.security.http.ssl.verification_mode: none should disable certificate verification but this is insecure so I wouldn't suggest it , and if you assumption that the env variables are not being read is true, then it will also not be read so it won't solve your problem.

3

Yes, you are correct. tls.crt is signed by this ca.crt.

For me it looks like the following bug:

elasticsearch-setup-passwords is ignoring the environment, but it is reading config/elasticsearch.yml.

I did following workaround:

I converted the output of xpack environment and converted it to json format. Then i added it on runtime to elasticsearch.yml. Then I ran the password script and I was able to connect correctly. Then (after some password-reset issue (somehow password of elastic had been set before and it did not match the bootstrap password) I have been able to run the script and set the passwords.

So I think reading config from environment is not working with that script.

4

I'm not sure I follow this. How did you convert the settings to json format and then added to the YAML file ?

Could you give us a little bit more information to try and replicate that issue ?

5

Sorry, I mixed json and yaml. I converted to yaml.
As you saw above my environment setting looks like this:

xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/http/ca.crt
xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/transport/ca.crt
xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/http/tls.crt

My conversion is just replacing = by : and puts the value into doublequotes:

xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/http/ca.crt"
xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/transport/ca.crt"
xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/http/tls.crt"

nope, just using plain kubernetes.

yes

I just created ca and certs with elasticsearch-certutil in pem format.
Container configuration is completely done by environment.

most part of kubernetes manifest can be found here:
https://pastebin.com/3LRBr5vN

Hope that helps to reproduce it.

Regards, Andreas

6

Awesome, thanks for the details ! We'll take a look and I'll update here with what we figured out

7

That is correct.

For docker we have a special custom entry point that converts all the environment variables into -E options to Elasticsearch.
That is a special property of the docker image and intentionally isn't used by our general scripts.

Your options to get this working are either:

  1. Pass the relevant settings as -E options to elasticsearch-setup-passwords
  2. Clone the above part of the docker-entrypoint script and then put it into a wrapper around elasticsearch-setup-passwords so it automatically pulls in the relevant environment variables.

With pure docker we've generally expected that people won't need to run scripts in the container that have this environment reading magic, but that assumption seems to be less true these days, so we may need to look at something that can replicate that behaviour in our other utilities.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.