# Client SSL Handshake Issues **URL:** \ **Category:** Elasticsearch\ **Created:** [January 11, 2018, 7:57pm UTC](https://discuss.elastic.co/t/client-ssl-handshake-issues/115167 "2018-01-11T19:57:49Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Senor](https://avatars.discourse-cdn.com/v4/letter/s/5f8ce5/32.png) [@Senor](https://discuss.elastic.co/u/Senor)\ **Post date:** [January 11, 2018, 7:57pm UTC](https://discuss.elastic.co/t/client-ssl-handshake-issues/115167/1 "2018-01-11T19:57:50Z")
I am running an Elasticsearch 6.1.1 cluster with 9 nodes on Ubuntu 16.04.3 running JVM 9. One node is a dedicated master, two more are master-eligible and data, and the rest are data nodes. I have setup SSL by following along the Elastic documentation, and added additional settings to my **elasticsearch.yml** file that ended up being necessary for transport communication to work. I generated my own CA using **certutil** and generated all the certs for each server using **certutil**. The transport layer works successfully, and the cluster status when looking at the log file (_/var/log/elasticsearch/_) is green. Everything is successfully communicating. However, when I attempt to perform any client-operations (such as \_curl [https://ES-MASTER-01:9300/\_](https://ES-MASTER-01:9300/_)), I receive the following error: > curl: (35) gnutls\_handshake() failed: Certificate is bad If I try to hit port 9200 (_curl [https://ES-MASTER-01:9200](https://ES-MASTER-01:9200)_), I receive another error: > curl: (51) SSL: certificate subject name (ES-MASTER-01) does not match target host name 'ES-MASTER-01' This message is even further confusing given that the two explicitly match one another, at least in this output. I know that Elasticsearch has a requirement of the certificates allowing both serverAuth and clientAuth, which **certutil** presumably performs. Verification of the handshake using an OpenSSL command shows failure: > openssl s\_client -connect ES-MASTER-01:9300 | openssl x509 -text -noout ``` depth=1 CN = MyOrganizationES Global CA verify return:1 depth=0 CN = ES-MASTER-01 verify return:1 140281558685336:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1487:SSL alert number 42 140281558685336:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: Certificate: ``` I have added the CA to each server's key store as well as JVM's (method outlined below). The required keys are within the Elastic config directory as specified by the documentation (_/etc/elasticsearch/x-pack_). My **elasticsearch.yml** config file on all nodes is similar to such: ``` xpack.ssl.key: /etc/elasticsearch/x-pack/ES-MASTER-01.key xpack.ssl.key_passphrase: xpack.ssl.certificate: /etc/elasticsearch/x-pack/ES-MASTER-01.crt xpack.ssl.certificate_authorities: ["/etc/elasticsearch/x-pack/ca.crt"] xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.http.ssl.enabled: true xpack.security.audit.enabled: true xpack.security.enabled: true ``` Below is the method in which I used to generate the CA and server keys, and other relevant operations. ``` # Generate the certificates. This was performed on ES-MASTER-01. cd /usr/share/elasticsearch # Generate the CA. bin/x-pack/certutil ca --ca-dn 'CN=MyOrganizationES Global CA' --pass --days 3650 --keysize 4096 --out MyOrganizationES_CA.zip --pem # Copy the ca.key and ca.crt to /usr/share/elasticsearch/ca/ # Generate the instance key pairs. I used a valid YAML file with name, DNS, and IP. bin/x-pack/certutil cert --ca-cert ca/ca.crt --ca-key ca/ca.key --ca-pass --days 3650 --in MyOrganizationES_Cluster.yml --keysize 4096 --out MyOrganizationES_Keys.zip --pass --pem # Copy the certificate files from the .zip file to corresponding server, as well as the ca.crt. /etc/elasticsearch/x-pack # Update the machine's certificate store (all nodes). # Copy the ca.crt file to this location. /usr/local/share/ca-certificates update-ca-certificates --fresh # Update Java's cacerts store (all nodes). keytool -import -trustcacerts -cacerts -storepass changeit -noprompt -alias MyOrganizationES -file /usr/local/share/ca-certificates/ca.crt ``` At this point I believe there to be something with OpenSSL that is not liking the self-signed certificates, but I'm not sure how to further diagnose. I am also having the issue of Kibana refusing to start with the basic SSL configuration, but I believe that might be a symptom of this larger issue. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 11, 2018, 8:18pm UTC](https://discuss.elastic.co/t/client-ssl-handshake-issues/115167/2 "2018-01-11T20:18:15Z")
Moved to #x-pack ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [January 12, 2018, 10:11am UTC](https://discuss.elastic.co/t/client-ssl-handshake-issues/115167/3 "2018-01-12T10:11:36Z")
> [@Senor](#): > > curl [https://ES-MASTER-01:9300/](https://ES-MASTER-01:9300/) If you really mean that you're trying to access the "transport" port over HTTP, that's never going to work. It's a custom binary protocol, it doesn't understand HTTP. > [@Senor](#): > > curl: (51) SSL: certificate subject name (ES-MASTER-01) does not match target host name 'ES-MASTER-01' Can you run the cert file through ```auto openssl x509 -text -noout -in ES-MASTER-01.crt ``` and post the result. It sounds like the cert has incorrect SAN fields, which shouldn't happen, but it depends on what's in your `MyOrganizationES_Cluster.yml` ---
**Author:** ![Senor](https://avatars.discourse-cdn.com/v4/letter/s/5f8ce5/32.png) [@Senor](https://discuss.elastic.co/u/Senor)\ **Post date:** [January 12, 2018, 4:39pm UTC](https://discuss.elastic.co/t/client-ssl-handshake-issues/115167/4 "2018-01-12T16:39:56Z")
Hi Tim, Thank you for the info regarding 9300. I misunderstood it's function and thought it was similar to 80/443 (9200 for http, 9300 for https). There was actually a mistake in my YAML file. I failed to explicitly specify the host name within the "dns" section and only included the FQDN. **Incorrect:** ``` instances: - name: "ES-MASTER-01" ip: - "10.20.20.20" dns: - "ES-MASTER-01.domain.com" - name: "ES-DATA-01" ip: - "10.20.20.21" dns: - "ES-DATA-01.domain.com" ... ``` **Correct:** ``` instances: - name: "ES-MASTER-01" ip: - "10.20.20.20" dns: - "ES-MASTER-01" - "ES-MASTER-01.domain.com" - name: "ES-DATA-01" ip: - "10.20.20.21" dns: - "ES-DATA-01" - "ES-DATA-01.domain.com" ... ``` This resolved the handshake issues. Unfortunately when I do _curl_ I receive an authentication error even when I explicitly pass the **elastic** user password. It might be by design once X-Pack security is enabled, in which case, it's a non-issue. ``` $ curl https://ES-MASTER-01:9200/_cluster/health?pretty -u elastic: { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "failed to authenticate user [elastic]", "header" : { "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\"" } } ], "type" : "security_exception", "reason" : "failed to authenticate user [elastic]", "header" : { "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\"" } }, "status" : 401 } ``` I'm also still having issues with Kibana failing to start at all with SSL enabled, and unfortunately it does not log anything at all to _/var/log/kibana/kibana.log_. These are the settings I have within _kibana.yml_ ``` server.host: "10.20.20.20" server.name: "MyOrganizationKibana" elasticsearch.url: "https://10.20.20.20:9200" elasticsearch.username: "kibana" elasticsearch.password: "" server.ssl.enabled: true server.ssl.certificate: /etc/elasticsearch/x-pack/ES-MASTER-01.crt server.ssl.key: /etc/elasticsearch/x-pack/ES-MASTER-01.key server.ssl.key_passphrase: elasticsearch.ssl.certificate: /etc/elasticsearch/x-pack/ES-MASTER-01.crt elasticsearch.ssl.key: /etc/elasticsearch/x-pack/ES-MASTER-01.key elasticsearch.ssl.key_passphrase: elasticsearch.ssl.certificateAuthorities: ["/etc/elasticsearch/x-pack/ca.crt"] logging.dest: /var/log/kibana/kibana.log xpack.security.encryptionKey: "" xpack.security.sessionTimeout: 600000 ``` If I disable SSL within Kibana, it does start normally and the authentication page loads with an error which I assume is because it is failing to authenticate with the cluster: ``` Login is currently disabled. Administrators should consult the Kibana logs for more details. ``` Kibana stays initialized and writes the the log file. After initializing its plugins, it goes from [green] to [yellow] to [red]. Based on what I understand of the logs, its not able to get a proper handshake which makes sense given that SSL isn't specified. ``` {"type":"log","@timestamp":"2018-01-12T15:52:50Z","tags":["error","elasticsearch","admin"],"pid":2677,"message":"Request error, retrying\nHEAD https://10.20.20.20:9200/ => unable to verify the first certificate"} ``` After it fully fails its initialization, It repeats this error message endlessly with no other log entries aside from these: ``` {"type":"ops","@timestamp":"2018-01-12T15:53:04Z","tags":[],"pid":2677,"os":{"load":[0.18017578125,0.12890625,0.21923828125],"mem":{"total":67559706624,"free":47786881024},"uptime":57987},"proc":{"uptime":24.188,"mem":{"rss":346910720,"heapTotal":303140864,"heapUsed":263092720,"external":6084687},"delay":0.41111599653959274},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"concurrents":{"5601":6},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 250.9MB uptime: 0:00:24 load: [0.18 0.13 0.22] delay: 0.411"} {"type":"log","@timestamp":"2018-01-12T15:53:07Z","tags":["warning","elasticsearch","admin"],"pid":2677,"message":"Unable to revive connection: https://10.20.20.20:9200/"} {"type":"log","@timestamp":"2018-01-12T15:53:07Z","tags":["warning","elasticsearch","admin"],"pid":2677,"message":"No living connections"} {"type":"log","@timestamp":"2018-01-12T15:53:09Z","tags":["warning","elasticsearch","admin"],"pid":2677,"message":"Unable to revive connection: https://10.20.20.20:9200/"} {"type":"log","@timestamp":"2018-01-12T15:53:09Z","tags":["warning","elasticsearch","admin"],"pid":2677,"message":"No living connections"} {"type":"ops","@timestamp":"2018-01-12T15:53:09Z","tags":[],"pid":2677,"os":{"load":[0.16552734375,0.12646484375,0.2177734375],"mem":{"total":67559706624,"free":47786180608},"uptime":57992},"proc":{"uptime":29.188,"mem":{"rss":347451392,"heapTotal":303140864,"heapUsed":263746056,"external":6168207},"delay":0.3634580001235008},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"concurrents":{"5601":6},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 251.5MB uptime: 0:00:29 load: [0.17 0.13 0.22] delay: 0.363"} {"type":"log","@timestamp":"2018-01-12T15:53:12Z","tags":["warning","elasticsearch","admin"],"pid":2677,"message":"Unable to revive connection: https://10.20.20.20:9200/"} {"type":"log","@timestamp":"2018-01-12T15:53:12Z","tags":["warning","elasticsearch","admin"],"pid":2677,"message":"No living connections"} {"type":"log","@timestamp":"2018-01-12T15:53:14Z","tags":["warning","elasticsearch","admin"],"pid":2677,"message":"Unable to revive connection: https://10.20.20.20:9200/"} {"type":"log","@timestamp":"2018-01-12T15:53:14Z","tags":["warning","elasticsearch","admin"],"pid":2677,"message":"No living connections"} ``` The Elasticsearch logs (_/var/log/elasticsearch/MyOrganizationES.log_) show no events that correspond to these requests either, unfortunately. ---
**Author:** ![Senor](https://avatars.discourse-cdn.com/v4/letter/s/5f8ce5/32.png) [@Senor](https://discuss.elastic.co/u/Senor)\ **Post date:** [January 12, 2018, 8:31pm UTC](https://discuss.elastic.co/t/client-ssl-handshake-issues/115167/5 "2018-01-12T20:31:58Z")
Quick update. The _curl_ issue was my own doing. I thought I had already generated our passwords, but I now recall that when we moved to 6.1.1, I started from scratch and created a fresh cluster on newly provisioned VMs. It seems I somehow skipped over password generation. That was a brain fart on my part. I successfully ran _bin/x-pack/setup-passwords **interactive** _ and am now able to _curl_ the cluster by passing -u _elastic:pass_ without any issue. I am still troubleshooting the Kibana issue but have not made any headway. ---
**Author:** ![Senor](https://avatars.discourse-cdn.com/v4/letter/s/5f8ce5/32.png) [@Senor](https://discuss.elastic.co/u/Senor)\ **Post date:** [January 15, 2018, 8:04pm UTC](https://discuss.elastic.co/t/client-ssl-handshake-issues/115167/6 "2018-01-15T20:04:17Z")
I was able to get Kibana to load with SSL enabled after I copied the certificates to _/etc/kibana/_ and gave explicit ownership to the _kibana_ user (chown). Unfortunately, I am still receiving the looping error message about "No living connections". ``` "type":"log","@timestamp":"2018-01-15T17:59:43Z","tags":["license","warning","xpack"],"pid":2669,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"} {"type":"ops","@timestamp":"2018-01-15T17:59:44Z","tags":[],"pid":2669,"os":{"load":[0.017578125,0.11376953125,0.20849609375],"mem":{"total":67559706624,"free":46817746944},"uptime":1630},"proc":{"uptime":216.778,"mem":{"rss":164679680,"heapTotal":125018112,"heapUsed":107124456,"external":1686457},"delay":0.18711599987000227},"load":{"requests":{},"concurrents":{"5601":0},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 102.2MB uptime: 0:03:37 load: [0.02 0.11 0.21] delay: 0.187"} {"type":"log","@timestamp":"2018-01-15T17:59:46Z","tags":["warning","elasticsearch","admin"],"pid":2669,"message":"Unable to revive connection: https://10.20.20.20:9200/"} {"type":"log","@timestamp":"2018-01-15T17:59:46Z","tags":["warning","elasticsearch","admin"],"pid":2669,"message":"No living connections"} ``` If I curl the cluster using the _kibana_ user, I am able to authenticate properly: ``` $ curl https://10.20.20.20:9200/_cluster/health?pretty -u kibana: { "cluster_name" : "MyOrganizationES", "status" : "green", "timed_out" : false, "number_of_nodes" : 9, "number_of_data_nodes" : 8, "active_primary_shards" : 2348, "active_shards" : 2392, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` The updated **kibana.yml** settings are as follows: ``` server.host: "ES-MASTER-01" server.name: "MyOrganizationKibana" elasticsearch.url: "https://10.20.20.20:9200" elasticsearch.username: "kibana" elasticsearch.password: "" server.ssl.enabled: true server.ssl.certificate: /etc/kibana/ES-MASTER-01.crt server.ssl.key: /etc/kibana/ES-MASTER-01.key server.ssl.keyPassphrase: elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/ES-MASTER-01.crt"] logging.dest: /var/log/kibana/kibana.log logging.verbose: true xpack.security.enabled: true xpack.security.encryptionKey: "" xpack.security.sessionTimeout: 600000 ``` Any assistance would be greatly appreciated. I'm at a loss here. ---
**Author:** ![Senor](https://avatars.discourse-cdn.com/v4/letter/s/5f8ce5/32.png) [@Senor](https://discuss.elastic.co/u/Senor)\ **Post date:** [January 16, 2018, 4:43pm UTC](https://discuss.elastic.co/t/client-ssl-handshake-issues/115167/7 "2018-01-16T16:43:40Z")
In case anyone ever comes across this on a Google, I did find a solution. It was a mistake on my part. For `elasticsearch.ssl.certificateAuthorities` in `kibana.yml`, I was invalidly referencing the node's certificate instead of the actual CA cert file. ``` # Correct: elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/ca.crt"] # Incorrect: elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/ES-MASTER-01.crt"] ``` I'm kicking myself over that one, but both Elasticsearch and Kibana are working successfully now with SSL. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 13, 2018, 4:43pm UTC](https://discuss.elastic.co/t/client-ssl-handshake-issues/115167/8 "2018-02-13T16:43:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.