Elasticsearch shield SSL nodes unavailable

I am trying to connect to elastic search 2.1.1 using transport client by setting up Shield SSL. I have succeeded in creating the certificate and the keystore as mentioned in the Shield SSL set up guide using OpenSSL and java keytool. I do not see any certificate/security issues while the server starts.

I am able to connect to elastic search with basic shield authentication, but once I turn on the SSL settings, I am getting NoNodeAvailableException.

Exception in thread "main" NoNodeAvailableException[None of the configured nodes
are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9301}]]

Here are my elasticsearch.yml settings

network.host: 127.0.0.1
cluster.name: es_raaj
node.name: wolverine_dev
shield.ssl.keystore.path:C:/elasticsearch-2.1.1/config/shield/wolverine_dev.jks
shield.ssl.keystore.password:
shield.ssl.keystore.key_password:
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300"]
shield.transport.ssl: true

here is my code

Settings settings = Settings.settingsBuilder()
                                        .put("cluster.name", "es_raaj")
                                        .put("node.name","wolverine_dev")
                                        .put("shield.transport.ssl", true)
                                        .put("shield.user", "romanov:<password>")
                                        .put("shield.ssl.keystore.path", "wolverine_dev.jks")
                                        .put("shield.ssl.keystore.password", "<password>")
                                        //.put("action.bulk.compress", false)
                                        //.put("client.transport.sniff",true)
                                        .build();
            client = TransportClient.builder()
                            .addPlugin(ShieldPlugin.class)
                            .settings(settings).build()
                            .addTransportAddress(new 
InetSocketTransportAddress(InetAddress.getByName("localhost"), 9300));

I have tried various settings for the transport client which I have commented above
Below is my elasticsearch server start up log, if it helps

[2016-06-10 16:48:30,872][INFO ][node                     ] [wolverine_dev] version[2.1.1], pid[17568], build[40e2c53/2015-12-15T13:05:55Z]
[2016-06-10 16:48:30,872][INFO ][node                     ] [wolverine_dev] initializing ...
[2016-06-10 16:48:31,238][INFO ][plugins                  ] [wolverine_dev] loaded [shield, license], sites []
[2016-06-10 16:48:31,269][INFO ][env                      ] [wolverine_dev] using [1] data paths, mounts [[OSDisk (C:)]], net usable_space [16gb], net total_space [118.7gb], spins? [unknown], types [N
TFS]
[2016-06-10 16:48:31,571][INFO ][http                     ] [wolverine_dev] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by [shield]
[2016-06-10 16:48:31,834][INFO ][transport                ] [wolverine_dev] Using [org.elasticsearch.shield.transport.ShieldServerTransportService] as transport service, overridden by [shield]
[2016-06-10 16:48:31,835][INFO ][transport                ] [wolverine_dev] Using [org.elasticsearch.shield.transport.netty.ShieldNettyTransport] as transport, overridden by [shield]
[2016-06-10 16:48:34,259][INFO ][node                     ] [wolverine_dev] initialized
[2016-06-10 16:48:34,259][INFO ][node                     ] [wolverine_dev] starting ...
[2016-06-10 16:48:35,641][INFO ][shield.transport         ] [wolverine_dev] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2016-06-10 16:48:35,652][INFO ][discovery                ] [wolverine_dev] es_raaj/uts9wTn6RRi_XQwdZCoKqQ
[2016-06-10 16:48:39,777][INFO ][cluster.service          ] [wolverine_dev] new_master {wolverine_dev}{uts9wTn6RRi_XQwdZCoKqQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0]
 joins received)
[2016-06-10 16:48:39,922][INFO ][license.plugin.core      ] [wolverine_dev] license [048bf56b-6c1a-46fb-8857-2809b00325a1] - valid
[2016-06-10 16:48:39,927][ERROR][license.plugin.core      ] [wolverine_dev]
[2016-06-10 16:48:39,981][INFO ][gateway                  ] [wolverine_dev] recovered [1] indices into cluster_state
[2016-06-10 16:48:40,000][INFO ][http                     ] [wolverine_dev] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2016-06-10 16:48:40,001][INFO ][node                     ] [wolverine_dev] started

I have verified the jar versions of elastic search and shield on both my client and server, its 2.1.1
Please help me resolving this issue.

Can you set the node.name setting to something different for the transport client?

I only have one node on my cluster.

Right, but the transport client will use the same name as your node because you define it in the settings.

Also in your exception, the port is 9301 but the settings shown use 9300, is this from different attempts?

Oh that one.. yeah its from multiple attempts. I matched the port number when it changed on starting my server.
I am new to elastic search, so how do I set a different node name for a TransportClient?

anyone has any suggestions on this? been stuck on this for a long time. not sure what am I missing.

How does Elastic Search's nodes work when an SSL layer is added using shield.?

From the server you are running Elasticsearch on what does:

openssl s_client -connect 127.0.0.1:9300 | openssl x509 -text -noout

output? Also what does:
ss -ltpn |grep java

output?

Thank you.. here is the output

c:\OpenSSL-Win64\bin>openssl s_client -connect 127.0.0.1:9300 | openssl x509 -text -noout
depth=1 O = DTE Energy, OU = CNI Team, emailAddress = raaj@DTE.COM, L = Detroit, ST = Michigan, C = US, CN = Raaj Mandal
verify error:num=19:self signed certificate in certificate chain
12596:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:.\ssl\s3_pkt.c:1472:SSL alert number 42
12596:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:.\ssl\s23_lib.c:177:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2 (0x2)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=DTE Energy, OU=CNI Team/emailAddress=raaj@DTE.COM, L=Detroit, ST=Michigan, C=US, CN=Raaj Mandal
Validity
Not Before: Jun 10 18:04:14 2016 GMT
Not After : May 23 18:04:14 2018 GMT
Subject: CN=.
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:94:b6:e7:19:45:9f:b8:7e:33:86:2a:38:fe:e8:
c6:d8:18:f9:31:be:7d:ef:0d:a2:66:51:bf:7d:c8:
ce:9b:42:0f:92:da:24:04:9b:56:59:b5:f7:d4:10:
4f:97:66:34:d0:f7:6f:55:aa:79:8d:4a:18:84:85:
70:6a:e5:c5:05:0d:92:ea:77:8c:4a:f0:65:3c:c0:
8a:76:73:b0:f3:8e:0e:3a:f7:03:92:67:79:06:cb:
ec:cb:4e:83:d7:f8:b0:74:43:8a:ba:17:6e:4f:77:
61:5c:bf:e5:79:5f:a1:ae:a3:c0:59:05:3f:e0:8e:
df:20:32:89:c0:c8:19:3a:8b:1d:42:d8:52:19:19:
ab:52:87:f1:6a:78:b3:ef:f4:18:1e:c1:22:df:e2:
2a:0c:5b:c0:f8:43:70:57:33:1f:93:df:1f:73:8e:
68:b9:23:9b:a8:fb:b8:2d:be:0d:45:bc:ae:e3:6c:
f8:c3:9c:6f:98:85:3e:7c:4a:41:42:a6:de:81:37:
38:f9:42:a5:d6:31:9f:33:e7:08:ad:6e:3e:c3:8f:
c5:f1:f1:dc:c4:16:57:7b:8d:bb:be:6e:0f:38:4f:
6e:30:e2:8b:68:23:31:fa:2b:24:7f:63:ed:f9:2c:
83:60:db:28:15:31:f0:18:5c:ed:66:41:e6:5a:ed:
3f:63
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
3B:7A:EE:30:56:35:12:7E:F9:0D:24:E0:42:D3:C0:35:0C:ED:25:03
X509v3 Subject Alternative Name:
IP Address:127.0.0.1
Signature Algorithm: sha256WithRSAEncryption
68:d5:15:db:6d:9a:56:e1:62:3d:98:a4:b1:5a:c8:6a:ea:3d:
01:77:3c:d5:e9:fd:bd:4e:c1:2b:4d:b8:99:66:75:8f:bb:2c:
ef:69:ad:c5:f3:50:7a:47:13:3c:4e:9b:84:e8:40:0b:38:7c:
0c:73:72:11:0b:ea:4a:24:92:dd:ed:67:3c:98:6a:8d:bb:96:
f0:96:c7:c5:84:1a:02:52:e1:02:06:31:d1:18:f5:11:fb:fb:
32:a1:cd:c4:43:2f:0b:fd:28:20:b7:0b:7f:00:31:62:04:88:
92:94:65:05:25:8b:bc:d7:b1:ce:ed:c4:b4:93:7d:35:e6:27:
21:0d:9a:81:40:48:bd:99:70:da:65:39:4b:36:04:35:e1:8b:
49:6e:2c:3b:c7:97:aa:0e:29:a8:63:5a:0b:90:4b:71:31:81:
ae:15:3b:c6:df:0f:62:46:3a:43:43:0d:b3:f2:27:47:0e:ba:
7f:22:4e:5d:8e:d6:3d:ff:31:83:0b:d1:54:61:25:44:d9:c4:
fd:96:1a:a7:a0:40:60:c7:c6:fe:3e:d4:17:d0:41:e7:18:f1:
2a:74:fc:f9:17:49:da:a2:00:d8:be:38:a2:8f:b3:4d:46:df:
9f:7d:df:19:16:15:ab:bc:80:10:b6:1c:36:79:c3:90:60:51:
2e:25:e8:89

This is the output of the first command.
I am not sure about the second command, does that work in windows?

Raaj

also this is the log from my cluster. I am assuming the self signed CA certificate is not working.

[2016-06-14 16:59:32,771][WARN ][shield.transport.netty ] [wolverine_dev] exception caught on transport layer [[id: 0x0ab49f4c, /127.0.0.1:60044 => /127.0.0.1:9300]
javax.net.ssl.SSLHandshakeException: null cert chain
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1336)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:796)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:764)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.ipfilter.IpFilteringHandlerImpl.handleUpstream(IpFilteringHandlerImpl.java:154)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1639)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:281)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:269)
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1638)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:176)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:901)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:841)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:839)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1273)
at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1392)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255)
... 21 more

Do you have access to an internal certificate authority that can sign your certificate request?

Yeah we do.. Another team handles all that stuff. They usually handle higher environment SSL certificates. is there anything in particular that I should specify if in case they are willing to provide me a certificate for my local testing environment?

You'll want to get the CA certificate to install on any elasticsearch or client server in addition to your signed certificate when you submit that case.

Okay so I get the certificate from them. I have my self signed certificate. The which certificate should I use to generate the jks file ? ?

Follow all the steps for making a jks file here:

https://www.elastic.co/guide/en/shield/current/ssl-tls.html

It will have your private key, the signed csr back from the ca, and the ca certificate in it when all the steps are complete.

Okay I got a CA signed certificate from Comodo.

C:\OpenSSL-Win64\bin>openssl s_client -connect vd-w7-122.dtenet.com:9301 | openssl x509 -text -noout

11052:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:.\ssl\s23_clnt.c:769:
unable to load certificate
6972:error:0906D06C:PEM routines:PEM_read_bio:no start line:.\crypto\pem\pem_lib.c:701:Expecting: TRUSTED CERTIFICATE

But I get this exception in the server log when I run the above command. Is this an issue with the certificate?

javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1336)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:796)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:764)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.ipfilter.IpFilteringHandlerImpl.handleUpstream(IpFilteringHandlerImpl.java:154)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1639)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:281)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:269)
at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:901)
at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:629)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:167)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:901)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:841)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:839)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1273)
at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1392)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255)
... 21 more

I also installed the intermediate certificates related to the signed CA certificate. I am still getting the above exceptions on starting up the server.

Are shield.transport.ssl: true and shield.http.ssl: true set?

Yes
network.host: 172.27.213.97
shield.transport.ssl: true
shield.ssl.truststore.path: node01.jks
shield.ssl.truststore.password: esadmin
shield.ssl.keystore.path: node01.jks
shield.ssl.keystore.password: esadmin
shield.ssl.keystore.key_password: esadmin
shield.http.ssl: true

In the above settings, I pointed trust store to the same jks file for testing purposes. I tested it without those settings also
I also tried this, I specified this DNS in SAN -ext, but when I set it here, and start the server I get a handshake exception if I do not apply the below zen discovery settings. This DNS is bound to this IP in my network.
#network.host: vd-w7-122.dtenet.com
I have commented these out, as I was testing all the settings, but none worked. #discovery.zen.ping.multicast.enabled: false
#discovery.zen.ping.unicast.hosts: ["172.27.213.97"]

Have you tried with the full path to the jks file?

Yes... this was how I used it
shield.ssl.keystore.path: C:/elasticsearch-2.1.1/config/shield/ca/node01.jks

by the way, I placed that .jks file in config folder, as it was looking for the jks file in that folder when I just specified the file name