Handshake failure (TransportClientNodesService: failed to connect to node)

I'm a little stuck figuring out why I'm getting a handshake failure when trying to connect to ES. I ran ES with the following command:

docker run -e ES_JAVA_OPTS='-Xms1g -Xmx1g' --name elasticsearch -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "xpack.security.enabled=false" -d -p 9200:9200 -p 9300:9300 docker.elastic.co/elasticsearch/elasticsearch:5.2.2

When I access ES from the browser at localhost:9200, I get:

"name": "zW-D1Xo",
"cluster_name": "docker-cluster",
"cluster_uuid": "j9DGTipFSB6qC5P-LWvxPQ",
"version": {
"number": "5.2.2",
"build_hash": "f9d9b74",
"build_date": "2017-02-24T17:26:45.835Z",
"build_snapshot": false,
"lucene_version": "6.4.1"
},
"tagline": "You Know, for Search"
}

I set up the TransportClient like so:

transportClient = new PreBuiltTransportClient(settingsBuilder.build())
          .addTransportAddresses(transportAddresses);

The only setting is cluster.name.

The error I get is this:

INFO  [2017-04-18 21:38:31,593] org.elasticsearch.client.transport.TransportClientNodesService: failed to connect to node [{#transport#-1}{oFK78nJYT9-FGrQbHEk-qg}{localhost}{127.0.0.1:9300}], removed from nodes list
! org.elasticsearch.transport.TransportException: connection reset
! at org.elasticsearch.transport.TcpTransport.onChannelClosed(TcpTransport.java:1610)
! at org.elasticsearch.transport.netty4.Netty4Transport.access$100(Netty4Transport.java:93)
! at org.elasticsearch.transport.netty4.Netty4Transport$ChannelCloseListener.operationComplete(Netty4Transport.java:412)
! at org.elasticsearch.transport.netty4.Netty4Transport$ChannelCloseListener.operationComplete(Netty4Transport.java:402)
! at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
! at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
! at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
! at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
! at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
! at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1058)
! at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:686)
! at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:664)
! at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:607)
! at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:82)
! at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:137)
! at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
! at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527)
! at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481)
! at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
! at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
! ... 1 common frames omitted
! Causing: java.lang.IllegalStateException: handshake failed
! at org.elasticsearch.transport.TcpTransport.executeHandshake(TcpTransport.java:1568)
! at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:502)
! ... 9 common frames omitted
! Causing: org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9300] general node connection failure
! at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:519)
! at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:460)
! at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318)
! at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:408)
! at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:354)
! at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:387)
! at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
! at java.lang.Thread.run(Thread.java:745)

Here is the Maven dependency (where dep.es.version is 5.2.2):

    <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>transport</artifactId>
      <version>${dep.es.version}</version>
      <exclusions>
        <exclusion>
          <groupId>joda-time</groupId>
          <artifactId>joda-time</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.httpcomponents</groupId>
          <artifactId>httpcore</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-logging</groupId>
          <artifactId>commons-logging</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${dep.es.version}</version>
      <exclusions>
        <exclusion>
          <groupId>joda-time</groupId>
          <artifactId>joda-time</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.httpcomponents</groupId>
          <artifactId>httpcore</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

I don't know what I should try next. Any ideas or directions I should pursue?

Thanks,
Benson

are you sure that transportAddresses are correct? and do you see an logs on the server?

I am just guessing since I am not familiar with this but specifying the same parameter twice looks suspicious?

Thanks for responding.

The transportAddresses are just localhost (I added a printout in my code below):

ADDRESS: [127.0.0.1:9300]
INFO  [2017-04-21 16:20:30,478] org.elasticsearch.client.transport.TransportClientNodesService: failed to connect to node [{#transport#-1}{8qMGuDVmR-WP94YLe4WqYg}{localhost}{127.0.0.1:9300}], removed from nodes list

As for the server, I'm not seeing anything on the elasticsearch side. Here are the total logs (this time I didn't load Kibana) in the docker container:

[2017-04-21T16:19:23,657][INFO ][o.e.n.Node               ] [] initializing ...
[2017-04-21T16:19:23,766][INFO ][o.e.e.NodeEnvironment    ] [4NynSsv] using [1] data paths, mounts [[/ (overlay)]], net usable_space [57.4gb], net total_space [62.7gb], spins? [possibly], types [overlay]
[2017-04-21T16:19:23,766][INFO ][o.e.e.NodeEnvironment    ] [4NynSsv] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-04-21T16:19:23,767][INFO ][o.e.n.Node               ] node name [4NynSsv] derived from node ID [4NynSsv9RHS8EYIvevF1vQ]; set [node.name] to override
[2017-04-21T16:19:23,771][INFO ][o.e.n.Node               ] version[5.2.2], pid[1], build[f9d9b74/2017-02-24T17:26:45.835Z], OS[Linux/4.9.21-moby/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_92-internal/25.92-b14]
[2017-04-21T16:19:25,370][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [aggs-matrix-stats]
[2017-04-21T16:19:25,370][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [ingest-common]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [lang-expression]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [lang-groovy]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [lang-mustache]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [lang-painless]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [percolator]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [reindex]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [transport-netty3]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [transport-netty4]
[2017-04-21T16:19:25,372][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded plugin [x-pack]
[2017-04-21T16:19:25,783][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
[2017-04-21T16:19:28,381][INFO ][o.e.n.Node               ] initialized
[2017-04-21T16:19:28,381][INFO ][o.e.n.Node               ] [4NynSsv] starting ...
[2017-04-21T16:19:28,653][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: 59:29:07:60:83:8c:94:96
[2017-04-21T16:19:28,730][INFO ][o.e.t.TransportService   ] [4NynSsv] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2017-04-21T16:19:31,826][INFO ][o.e.c.s.ClusterService   ] [4NynSsv] new_master {4NynSsv}{4NynSsv9RHS8EYIvevF1vQ}{_A8_cvJKQDirLQTHdntn7Q}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-04-21T16:19:31,857][INFO ][o.e.h.HttpServer         ] [4NynSsv] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2017-04-21T16:19:31,857][INFO ][o.e.n.Node               ] [4NynSsv] started
[2017-04-21T16:19:31,939][INFO ][o.e.g.GatewayService     ] [4NynSsv] recovered [0] indices into cluster_state
[2017-04-21T16:19:32,369][INFO ][o.e.l.LicenseService     ] [4NynSsv] license [e10d1dc3-f224-4806-bcc2-7dbef9219ddb] mode [trial] - valid
[2017-04-21T16:19:38,619][INFO ][o.e.c.m.MetaDataCreateIndexService] [4NynSsv] [.monitoring-es-2-2017.04.21] creating index, cause [auto(bulk api)], templates [.monitoring-es-2], shards [1]/[1], mappings [shards, _default_, node, index_stats, index_recovery, cluster_state, cluster_stats, indices_stats, node_stats]
[2017-04-21T16:19:38,733][INFO ][o.e.c.m.MetaDataCreateIndexService] [4NynSsv] [.monitoring-data-2] creating index, cause [auto(bulk api)], templates [.monitoring-data-2], shards [1]/[1], mappings [logstash, node, _default_, kibana, cluster_info]
[2017-04-21T16:19:38,983][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [cluster_stats]
[2017-04-21T16:19:39,037][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [node_stats]
[2017-04-21T16:19:49,185][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [index_stats]
[2017-04-21T16:19:49,266][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [cluster_stats]
[2017-04-21T16:19:49,319][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [indices_stats]

You can expose multiple ports:

Any other thoughts?

Thanks in advance.

I'm not sure if this is useful, but I tried curling the ports directly:

$ curl localhost:9200
{
  "name" : "4NynSsv",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "kOhN_OIrRoyAUc15sJEs7A",
  "version" : {
    "number" : "5.2.2",
    "build_hash" : "f9d9b74",
    "build_date" : "2017-02-24T17:26:45.835Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.1"
  },
  "tagline" : "You Know, for Search"
}
$ curl localhost:9300
curl: (52) Empty reply from server
$ curl localhost:9400
curl: (7) Failed to connect to localhost port 9400: Connection refused

I did the last one just to see if I would get a different reply for a invalid port.

hmm we usually try to detect if you are trying to talk to a tranport port via HTTP so you should get a reply with a line like this: This is not a HTTP port do you get this if you do curl -XGET http://localhost:9300?

I get the same error as before:

$ curl -XGET http://localhost:9300
curl: (52) Empty reply from server

when I do this I get this on 5.3

curl -XGET http://localhost:9300
This is not a HTTP port

So all I ran was the docker command:

docker run -e ES_JAVA_OPTS='-Xms1g -Xmx1g' --name elasticsearch -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "xpack.security.enabled=false" -d -p 9200:9200 -p 9300:9300 docker.elastic.co/elasticsearch/elasticsearch:5.2.2

and that curl. I'm on a Mac. Not sure if that makes a difference. I'm also using docker beta:

Version 17.05.0-ce-rc1-mac8 (16582)
Channel: edge
73d01bb48e

What are your specs?

OK, if I do the curl INSIDE the docker container, I get the same error that you get.

what if you set this to 0.0.0.0 as well?

That fixed everything. Thanks!!!!!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.