# Esrally-elastic mutual TLS **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [August 8, 2018, 12:37pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515 "2018-08-08T12:37:47Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![fara](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fara](https://discuss.elastic.co/u/fara)\ **Post date:** [August 8, 2018, 12:37pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/1 "2018-08-08T12:37:47Z")
Hi, I am trying to use rally(1.0.1) to connect to elastic 6.1 using mutual tls. I am completely sure the client certificates and key as well as CA cert are properly generated. Debugging the code ( **client.py** ) class I have found that **ssl\_context** is the one that seems to be "creating the issue". **Original code that does not work:** `return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection,ssl_context=self.ssl_context, **self.client_options)` with the next error: > urllib3.exceptions.SSLError: [SSL: SSLV3\_ALERT\_BAD\_CERTIFICATE] sslv3 alert bad certificate (\_ssl.c:645) **Modified code that works:** `return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection, **self.client_options)` Could anyone help me? Best Regards, Ramón. ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [August 8, 2018, 1:15pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/2 "2018-08-08T13:15:49Z")
Hi, can you please paste your Rally command line? Daniel ---
**Author:** ![fara](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fara](https://discuss.elastic.co/u/fara)\ **Post date:** [August 9, 2018, 6:56am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/3 "2018-08-09T06:56:59Z")
Hi, This is the command line I run: `esrally --track=pmc --target-hosts=10.10.10.10:31504 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,ca_certs:'/tmp/cacert.pem',client_cert:'/tmp/client_cert.pem',client_key:'/tmp/client_key.pem'"` Ramón. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [August 10, 2018, 3:04pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/4 "2018-08-10T15:04:43Z")
Hey @fara, Just to be on the safe side are you able to issue: ```auto curl --cacert /tmp/cacert.pem -u elastic: https://10.10.10.10:31504 ``` Also your `esrally` command is missing the username and password for Rally. It should look like: ```auto esrally --track=pmc --target-hosts=10.10.10.10:31504 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,ca_certs:'/tmp/cacert.pem',client_cert:'/tmp/client_cert.pem',client_key:'/tmp/client_key.pem',basic_auth_user:'user',basic_auth_password:'password'" ``` I quickly verified the above command against a simple [TLS enabled cluster in Docker](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls-docker.html) and it worked fine. Also note (depending on the SSL verification mode value on ES), that you don't always need to specify the client\_key. Dimitris ---
**Author:** ![fara](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fara](https://discuss.elastic.co/u/fara)\ **Post date:** [August 13, 2018, 6:46am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/5 "2018-08-13T06:46:58Z")
Hey @dliappis, I am not using X-Pack so we don't have user:password authentication enabled. We are just using Mutual TLS. According github documentation it should not be mandatory to use the user:pass authentication: > Enable SSL with a client key and certificate: `--client-options="use_ssl:true,verify_certs:true,ca_certs:'/path/to/cacert.pem',client_cert:'/path/to/client_cert.pem',client_key='/path/to/client_key.pem"` In rally version 0.9.2 was working without any basic authentication parameter Ramón. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [August 13, 2018, 10:24am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/6 "2018-08-13T10:24:54Z")
@fara I see. I presume you are having a different component (like `stunnel` or some proxy) in front of ES to enable TLS. Rally under the hood uses the [elasticsearch-py](http://elasticsearch-py.readthedocs.io/en/stable/) client and this is what handles the TLS part. Can you try passing the same certs as parameters in a curl command as mentioned [earlier](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/4?u=dliappis) and see if they work? No username/pass needed just a simple: ```auto curl --cacert /tmp/cacert.pem https://10.10.10.10:31504 ``` If this works the server isn't requesting client authentication; if it fails indicating client auth is needed, you can try instead: ```auto curl --cacert /tmp/cacert.pem --cert /tmp/client_cert.pem --key /tmp/client_key.pem https://10.10.10.10:31504 ``` This should help clarify the exact configuration and how to proceed further. ---
**Author:** ![fara](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fara](https://discuss.elastic.co/u/fara)\ **Post date:** [August 13, 2018, 10:32am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/7 "2018-08-13T10:32:43Z")
> [@fara](#): > > urllib3.exceptions.SSLError: [SSL: SSLV3\_ALERT\_BAD\_CERTIFICATE] sslv3 alert bad certificate (\_ssl.c:645) @dliappis I don't have any component before ES. It is just configured with Mutual TLS. I have done a tcpdump and I see that Elastic service is requiring the client certificate to rally but it is not sending them to ES. When I run: ```auto curl --cacert /tmp/cacert.pem --cert /tmp/client_cert.pem --key /tmp/client_key.pem https://10.10.10.10:31504 ``` I see the error mentioned in the first post. Ramón. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [August 13, 2018, 11:03am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/8 "2018-08-13T11:03:37Z")
I am not sure I understand the setup. To configure TLS for Elasticsearch (including [TLS for http clients](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html#tls-http)) you need [x-pack security](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html). If you are using Elasticsearch open source only and don't have any components in front of ES, what is requiring TLS? The `curl` error you got indicates there's some error with the certificates or their server config, so it makes sense to focus your investigation there. ---
**Author:** ![fara](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fara](https://discuss.elastic.co/u/fara)\ **Post date:** [August 13, 2018, 11:58am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/9 "2018-08-13T11:58:20Z")
Hey @dliappis Sorry if I exlained wrongly myself. As I said before we are not using X-Pack, but **search-guard** plugin to provide TLS configuration. As I said in the the first post, client.py class seems not to be sending the client certificate if the ssl\_context is enabled (which it is and can not be disabled by configuration). If I modify the client.py class from: `return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection,ssl_context=self.ssl_context, **self.client_options)` to: `return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection, **self.client_options)` I see that in the second case the client certificate is being send. Ramón ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [August 13, 2018, 12:17pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/10 "2018-08-13T12:17:57Z")
Ah I see. `search-guard` specific questions should be directed to the corresponding search-guard repo, but as mentioned earlier Rally uses the official elasticsearch-python client. As you can see [in the code](https://github.com/elastic/rally/blob/b7b032815120cf8ad77a483b4bdbc3c12015a7c5/esrally/client.py#L26-L32) there's nothing special done with `create_ssl_context` apart from passing the `ca_certs` parameter. Since you confirmed you are seeing the same error when you are passing the exact same parameters to `curl`, there's something wrong with either the cert or client side authentication. IIUC you are only interested in the encryption part and not the server authenticating the client, can you try **not verifying** the client certs? e.g. with curl, does this work? ```auto curl --cacert /tmp/cacert.pem https://10.10.10.10:31504 ``` If yes, can you try adjusting the Rally your `esrally` command to skip the verification using: ```auto esrally --track=pmc --target-hosts=10.10.10.10:31504 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,ca_certs:'/tmp/cacert.pem'" ``` ---
**Author:** ![fara](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fara](https://discuss.elastic.co/u/fara)\ **Post date:** [August 14, 2018, 8:01am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/11 "2018-08-14T08:01:54Z")
Hey @dliappis I have configured ES not to ask for client certificate and the command you typed works properly `esrally --track=pmc --target-hosts=10.10.10.10:31504 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,ca_certs:'/tmp/cacert.pem'"` From my understanding (I don't know too much python) but when the ES client is created in client.py the **client\_cert** and **client\_key** seems not to be passed to the ssl\_context when creating it. Moreover, with Mutual TLS activated in ES and when I run `curl https://10.10.10.10:31504/ -key clientkey.pem -cacert /tmp/cacert.pem -cert /tmp/clientcert.pem` everything works properly. Ramón. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [August 14, 2018, 9:56am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/12 "2018-08-14T09:56:38Z")
### For the `curl` observation: [Earlier](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/7?u=dliappis) you mentioned that with the same parameters (`curl https://10.10.10.10:31504/ -key clientkey.pem -cacert /tmp/cacert.pem -cert /tmp/clientcert.pem`) you were receiving the same bad certificate error as with Rally. Could you please double check that this curl command **does not** produce an error when Elasticsearch is configured requiring TLS verification from the client as well? ### For the `client_cert`/`client_key` question: TL;DR is no, when creating the `ssl_context`, the [allowable parameters](https://elasticsearch-py.readthedocs.io/en/stable/connection.html?highlight=connection#urllib3httpconnection-default-connection-class) are only the cafile in the form of [file, path or data](https://docs.python.org/3/library/ssl.html#ssl.create_default_context); in Rally we use [create\_ssl\_context](https://github.com/elastic/rally/blob/d285248a99dc16f1bd7f6003a766fb261e75abe0/esrally/client.py#L32) which is a [direct wrapper](https://github.com/elastic/elasticsearch-py/blob/cb6f09bbe74eca6cbf5bb2519c1f3d791c0dd820/elasticsearch/connection/http_urllib3.py#L27-L36) to [create\_default\_context](https://docs.python.org/3/library/ssl.html#ssl.create_default_context). The key and client certificate, along with any other python client parameters (defined in `--client-options`) such as `timeout`, get passed verbatim to the Python client as can be seen [here](https://github.com/elastic/rally/blob/b7b032815120cf8ad77a483b4bdbc3c12015a7c5/esrally/client.py#L14) and [here](https://github.com/elastic/rally/blob/b7b032815120cf8ad77a483b4bdbc3c12015a7c5/esrally/client.py#L91). ---
**Author:** ![fara](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fara](https://discuss.elastic.co/u/fara)\ **Post date:** [August 14, 2018, 12:09pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/13 "2018-08-14T12:09:12Z")
Hi, Thanks for the detailed explanation and sorry for the [Earlier](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/7) post , I was not precise enough. Catching up, I have ES configured with client authentication required (mutual tls). **For the Curl observation:** When I run the next curl against ES coordinator (coordinator-0-node.elasticsearch-4 which is resolvable in /etc/host) , and it works ( a json with the information of the coordinator is retrieve): > curl --key /tmp/client\_key.pem --cacert /tmp/cacert.pem --cert /tmp/client\_cert.pem [https://coordinator-0-node.elasticsearch-4:31504](https://coordinator-0-node.elasticsearch-4:31504) **For the esrally observation:** > esrally --track=pmc --target-hosts=coordinator-0-node.elasticsearch-4:31504 --pipeline=benchmark-only --client-options="use\_ssl:true,verify\_certs:false,ca\_certs:'/tmp/cacert.pem',client\_cert:'/tmp/client\_cert.pem',client\_key:'/tmp/client\_key.pem'" In the wireshark I see that the client certificate is not being sent during handshake process . Ramón. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [August 14, 2018, 8:45pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/14 "2018-08-14T20:45:05Z")
@fara thank for your latest comment, it _really_ clarified what you are trying to do and what's the issue. #### Facts: 1. You've configured ES (with a third party solution) to require TLS for the HTTP API **and** require verification of the client certs. 2. curl works when specifying the needed params (`--cacert`, `--cert`, `--key`). 3. Rally throws `:sslv3 alert bad certificate` even when specifying `client_cert`, `client_key`, `ca_certs` and requiring `verify_certs:true` in `--client-options`. 4. Rally (and curl) both work with HTTPS/TLS when the plugin doesn't require client verification. The facts above point to a bug, as you mentioned earlier. Apologies, it was necessary to go through the `curl` steps and understand completely what you are trying to achieve, to reach this conclusion. The good news is that I have been able to reproduce the issue using xpack:security and come up with a patch that fixes it. I will create a Rally bug, paste the link here and we can follow the discussion there. Thank you for your patience and help, Dimitris ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [August 15, 2018, 8:29am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/15 "2018-08-15T08:29:22Z")
Opened [https://github.com/elastic/rally/issues/550](https://github.com/elastic/rally/issues/550) ---
**Author:** ![fara](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fara](https://discuss.elastic.co/u/fara)\ **Post date:** [August 16, 2018, 6:22am UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/16 "2018-08-16T06:22:32Z")
Thank you very much for your help as well. Nice gist !!!!! ---
**Author:** ![fara](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fara](https://discuss.elastic.co/u/fara)\ **Post date:** [August 16, 2018, 1:06pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/17 "2018-08-16T13:06:06Z")
Hey @dliappis, I don't want to put myself where I don't belong and I pretty sure I might be wrong but I have seen your [PR](https://github.com/elastic/rally/pull/551) and from my understanding even if the SSL certificate verification is off, it should be possible to use mutuals TLS. Seeing your PR it seems that this is not possible. Ramón ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [August 16, 2018, 2:45pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/18 "2018-08-16T14:45:28Z")
Hey @fara, Feedback is welcome, let's continue the discussion in the GH issue, if you don't mind (you can just leave a comment in the PR: [https://github.com/elastic/rally/pull/551](https://github.com/elastic/rally/pull/551)) As a general comment, when verification is off, TLS can still be used and end2end communication will be encrypted. I am not 100% sure what third party plugins mean by the term Mutual TLS, but x-pack:security at least has an HTTP/TLS option to additionally control client authentication ([https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#http-tls-ssl-settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#http-tls-ssl-settings)); in this case it will try to verify the client certs, but it can be none or optional. So what I am trying to say is what's the meaning of "mutual" in TLS if it's not about verification, since when TLS is required on HTTP anyway the entire TCP flow between the client and ES server is encrypted? Regards, Dimitris ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [August 21, 2018, 3:22pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/19 "2018-08-21T15:22:50Z")
Fixes have been merged in the latest Rally master branch via [https://github.com/elastic/rally/pull/551](https://github.com/elastic/rally/pull/551). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 18, 2018, 3:22pm UTC](https://discuss.elastic.co/t/esrally-elastic-mutual-tls/143515/20 "2018-09-18T15:22:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.