Esrally-elastic mutual TLS

Elastic Stack Elasticsearch
1

Hi,

I am trying to use rally(1.0.1) to connect to elastic 6.1 using mutual tls.

I am completely sure the client certificates and key as well as CA cert are properly generated.

Debugging the code (client.py) class I have found that ssl_context is the one that seems to be "creating the issue".

Original code that does not work:
return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection,ssl_context=self.ssl_context, **self.client_options)

with the next error:

urllib3.exceptions.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:645)

Modified code that works:
return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection, **self.client_options)

Could anyone help me?

Best Regards, Ramón.

2

Hi,

can you please paste your Rally command line?

Daniel

3

Hi,

This is the command line I run:

esrally --track=pmc --target-hosts=10.10.10.10:31504 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,ca_certs:'/tmp/cacert.pem',client_cert:'/tmp/client_cert.pem',client_key:'/tmp/client_key.pem'"

Ramón.

4

Hey @fara,

Just to be on the safe side are you able to issue:

curl --cacert /tmp/cacert.pem -u elastic:<yourpassword> https://10.10.10.10:31504

Also your esrally command is missing the username and password for Rally. It should look like:

esrally --track=pmc --target-hosts=10.10.10.10:31504 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,ca_certs:'/tmp/cacert.pem',client_cert:'/tmp/client_cert.pem',client_key:'/tmp/client_key.pem',basic_auth_user:'user',basic_auth_password:'password'"

I quickly verified the above command against a simple TLS enabled cluster in Docker and it worked fine.

Also note (depending on the SSL verification mode value on ES), that you don't always need to specify the client_key.

Dimitris

5

Hey @dliappis,

I am not using X-Pack so we don't have user:password authentication enabled.

We are just using Mutual TLS.

According github documentation it should not be mandatory to use the user:pass authentication:

Enable SSL with a client key and certificate: --client-options="use_ssl:true,verify_certs:true,ca_certs:'/path/to/cacert.pem',client_cert:'/path/to/client_cert.pem',client_key='/path/to/client_key.pem"

In rally version 0.9.2 was working without any basic authentication parameter

Ramón.

6

@fara I see. I presume you are having a different component (like stunnel or some proxy) in front of ES to enable TLS. Rally under the hood uses the elasticsearch-py client and this is what handles the TLS part.

Can you try passing the same certs as parameters in a curl command as mentioned earlier and see if they work?
No username/pass needed just a simple:

curl --cacert /tmp/cacert.pem https://10.10.10.10:31504

If this works the server isn't requesting client authentication; if it fails indicating client auth is needed, you can try instead:

curl --cacert /tmp/cacert.pem --cert /tmp/client_cert.pem --key /tmp/client_key.pem https://10.10.10.10:31504

This should help clarify the exact configuration and how to proceed further.

7

@dliappis
I don't have any component before ES. It is just configured with Mutual TLS.
I have done a tcpdump and I see that Elastic service is requiring the client certificate to rally but it is not sending them to ES.
When I run:

curl --cacert /tmp/cacert.pem --cert /tmp/client_cert.pem --key /tmp/client_key.pem https://10.10.10.10:31504

I see the error mentioned in the first post.

Ramón.

8

I am not sure I understand the setup.

To configure TLS for Elasticsearch (including TLS for http clients) you need x-pack security.
If you are using Elasticsearch open source only and don't have any components in front of ES, what is requiring TLS?

The curl error you got indicates there's some error with the certificates or their server config, so it makes sense to focus your investigation there.

9

Hey @dliappis

Sorry if I exlained wrongly myself. As I said before we are not using X-Pack, but search-guard plugin to provide TLS configuration.

As I said in the the first post, client.py class seems not to be sending the client certificate if the ssl_context is enabled (which it is and can not be disabled by configuration).

If I modify the client.py class
from:
return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection,ssl_context=self.ssl_context, **self.client_options)

to:
return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection, **self.client_options)

I see that in the second case the client certificate is being send.

Ramón

10

Ah I see.

search-guard specific questions should be directed to the corresponding search-guard repo, but as mentioned earlier Rally uses the official elasticsearch-python client. As you can see in the code there's nothing special done with create_ssl_context apart from passing the ca_certs parameter. Since you confirmed you are seeing the same error when you are passing the exact same parameters to curl, there's something wrong with either the cert or client side authentication.

IIUC you are only interested in the encryption part and not the server authenticating the client, can you try not verifying the client certs?

e.g. with curl, does this work?

curl --cacert /tmp/cacert.pem https://10.10.10.10:31504

If yes, can you try adjusting the Rally your esrally command to skip the verification using:

esrally --track=pmc --target-hosts=10.10.10.10:31504 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,ca_certs:'/tmp/cacert.pem'"
11

Hey @dliappis

I have configured ES not to ask for client certificate and the command you typed works properly

esrally --track=pmc --target-hosts=10.10.10.10:31504 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,ca_certs:'/tmp/cacert.pem'"

From my understanding (I don't know too much python) but when the ES client is created in client.py the client_cert and client_key seems not to be passed to the ssl_context when creating it.

Moreover, with Mutual TLS activated in ES and when I run curl https://10.10.10.10:31504/ -key clientkey.pem -cacert /tmp/cacert.pem -cert /tmp/clientcert.pem everything works properly.

Ramón.

12

For the curl observation:

Earlier you mentioned that with the same parameters (curl https://10.10.10.10:31504/ -key clientkey.pem -cacert /tmp/cacert.pem -cert /tmp/clientcert.pem) you were receiving the same bad certificate error as with Rally.

Could you please double check that this curl command does not produce an error when Elasticsearch is configured requiring TLS verification from the client as well?

For the client_cert/client_key question:

TL;DR is no, when creating the ssl_context, the allowable parameters are only the cafile in the form of file, path or data; in Rally we use create_ssl_context which is a direct wrapper to create_default_context.

The key and client certificate, along with any other python client parameters (defined in --client-options) such as timeout, get passed verbatim to the Python client as can be seen here and here.

13

Hi,
Thanks for the detailed explanation and sorry for the Earlier post , I was not precise enough.

Catching up, I have ES configured with client authentication required (mutual tls).

For the Curl observation:
When I run the next curl against ES coordinator (coordinator-0-node.elasticsearch-4 which is resolvable in /etc/host) , and it works ( a json with the information of the coordinator is retrieve):

curl --key /tmp/client_key.pem --cacert /tmp/cacert.pem --cert /tmp/client_cert.pem https://coordinator-0-node.elasticsearch-4:31504

For the esrally observation:

esrally --track=pmc --target-hosts=coordinator-0-node.elasticsearch-4:31504 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,ca_certs:'/tmp/cacert.pem',client_cert:'/tmp/client_cert.pem',client_key:'/tmp/client_key.pem'"

In the wireshark I see that the client certificate is not being sent during handshake process .

Ramón.

14

@fara thank for your latest comment, it really clarified what you are trying to do and what's the issue.

Facts:

  1. You've configured ES (with a third party solution) to require TLS for the HTTP API and require verification of the client certs.
  2. curl works when specifying the needed params (--cacert, --cert, --key).
  3. Rally throws :sslv3 alert bad certificate even when specifying client_cert, client_key, ca_certs and requiring verify_certs:true in --client-options.
  4. Rally (and curl) both work with HTTPS/TLS when the plugin doesn't require client verification.

The facts above point to a bug, as you mentioned earlier. Apologies, it was necessary to go through the curl steps and understand completely what you are trying to achieve, to reach this conclusion.

The good news is that I have been able to reproduce the issue using xpack:security and come up with a patch that fixes it.

I will create a Rally bug, paste the link here and we can follow the discussion there.

Thank you for your patience and help,
Dimitris

15

Opened https://github.com/elastic/rally/issues/550

16

Thank you very much for your help as well.

Nice gist !!!!!

17

Hey @dliappis,

I don't want to put myself where I don't belong and I pretty sure I might be wrong but I have seen your PR and from my understanding even if the SSL certificate verification is off, it should be possible to use mutuals TLS. Seeing your PR it seems that this is not possible.

Ramón

18

Hey @fara,

Feedback is welcome, let's continue the discussion in the GH issue, if you don't mind (you can just leave a comment in the PR: https://github.com/elastic/rally/pull/551)

As a general comment, when verification is off, TLS can still be used and end2end communication will be encrypted. I am not 100% sure what third party plugins mean by the term Mutual TLS, but x-pack:security at least has an HTTP/TLS option to additionally control client authentication (https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#http-tls-ssl-settings); in this case it will try to verify the client certs, but it can be none or optional.

So what I am trying to say is what's the meaning of "mutual" in TLS if it's not about verification, since when
TLS is required on HTTP anyway the entire TCP flow between the client and ES server is encrypted?

Regards,
Dimitris

19

Fixes have been merged in the latest Rally master branch via https://github.com/elastic/rally/pull/551.

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.