Handshake failure (TransportClientNodesService: failed to connect to node)


#1

I'm a little stuck figuring out why I'm getting a handshake failure when trying to connect to ES. I ran ES with the following command:

docker run -e ES_JAVA_OPTS='-Xms1g -Xmx1g' --name elasticsearch -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "xpack.security.enabled=false" -d -p 9200:9200 -p 9300:9300 docker.elastic.co/elasticsearch/elasticsearch:5.2.2

When I access ES from the browser at localhost:9200, I get:

"name": "zW-D1Xo",
"cluster_name": "docker-cluster",
"cluster_uuid": "j9DGTipFSB6qC5P-LWvxPQ",
"version": {
"number": "5.2.2",
"build_hash": "f9d9b74",
"build_date": "2017-02-24T17:26:45.835Z",
"build_snapshot": false,
"lucene_version": "6.4.1"
},
"tagline": "You Know, for Search"
}

I set up the TransportClient like so:

transportClient = new PreBuiltTransportClient(settingsBuilder.build())
          .addTransportAddresses(transportAddresses);

The only setting is cluster.name.

The error I get is this:

INFO  [2017-04-18 21:38:31,593] org.elasticsearch.client.transport.TransportClientNodesService: failed to connect to node [{#transport#-1}{oFK78nJYT9-FGrQbHEk-qg}{localhost}{127.0.0.1:9300}], removed from nodes list
! org.elasticsearch.transport.TransportException: connection reset
! at org.elasticsearch.transport.TcpTransport.onChannelClosed(TcpTransport.java:1610)
! at org.elasticsearch.transport.netty4.Netty4Transport.access$100(Netty4Transport.java:93)
! at org.elasticsearch.transport.netty4.Netty4Transport$ChannelCloseListener.operationComplete(Netty4Transport.java:412)
! at org.elasticsearch.transport.netty4.Netty4Transport$ChannelCloseListener.operationComplete(Netty4Transport.java:402)
! at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
! at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
! at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
! at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
! at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
! at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1058)
! at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:686)
! at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:664)
! at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:607)
! at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:82)
! at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:137)
! at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
! at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527)
! at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481)
! at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
! at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
! ... 1 common frames omitted
! Causing: java.lang.IllegalStateException: handshake failed
! at org.elasticsearch.transport.TcpTransport.executeHandshake(TcpTransport.java:1568)
! at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:502)
! ... 9 common frames omitted
! Causing: org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9300] general node connection failure
! at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:519)
! at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:460)
! at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318)
! at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:408)
! at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:354)
! at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:387)
! at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
! at java.lang.Thread.run(Thread.java:745)

Here is the Maven dependency (where dep.es.version is 5.2.2):

    <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>transport</artifactId>
      <version>${dep.es.version}</version>
      <exclusions>
        <exclusion>
          <groupId>joda-time</groupId>
          <artifactId>joda-time</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.httpcomponents</groupId>
          <artifactId>httpcore</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-logging</groupId>
          <artifactId>commons-logging</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${dep.es.version}</version>
      <exclusions>
        <exclusion>
          <groupId>joda-time</groupId>
          <artifactId>joda-time</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.httpcomponents</groupId>
          <artifactId>httpcore</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

I don't know what I should try next. Any ideas or directions I should pursue?

Thanks,
Benson


(Simon Willnauer) #2

are you sure that transportAddresses are correct? and do you see an logs on the server?

I am just guessing since I am not familiar with this but specifying the same parameter twice looks suspicious?


#3

Thanks for responding.

The transportAddresses are just localhost (I added a printout in my code below):

ADDRESS: [127.0.0.1:9300]
INFO  [2017-04-21 16:20:30,478] org.elasticsearch.client.transport.TransportClientNodesService: failed to connect to node [{#transport#-1}{8qMGuDVmR-WP94YLe4WqYg}{localhost}{127.0.0.1:9300}], removed from nodes list

As for the server, I'm not seeing anything on the elasticsearch side. Here are the total logs (this time I didn't load Kibana) in the docker container:

[2017-04-21T16:19:23,657][INFO ][o.e.n.Node               ] [] initializing ...
[2017-04-21T16:19:23,766][INFO ][o.e.e.NodeEnvironment    ] [4NynSsv] using [1] data paths, mounts [[/ (overlay)]], net usable_space [57.4gb], net total_space [62.7gb], spins? [possibly], types [overlay]
[2017-04-21T16:19:23,766][INFO ][o.e.e.NodeEnvironment    ] [4NynSsv] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-04-21T16:19:23,767][INFO ][o.e.n.Node               ] node name [4NynSsv] derived from node ID [4NynSsv9RHS8EYIvevF1vQ]; set [node.name] to override
[2017-04-21T16:19:23,771][INFO ][o.e.n.Node               ] version[5.2.2], pid[1], build[f9d9b74/2017-02-24T17:26:45.835Z], OS[Linux/4.9.21-moby/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_92-internal/25.92-b14]
[2017-04-21T16:19:25,370][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [aggs-matrix-stats]
[2017-04-21T16:19:25,370][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [ingest-common]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [lang-expression]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [lang-groovy]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [lang-mustache]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [lang-painless]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [percolator]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [reindex]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [transport-netty3]
[2017-04-21T16:19:25,371][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded module [transport-netty4]
[2017-04-21T16:19:25,372][INFO ][o.e.p.PluginsService     ] [4NynSsv] loaded plugin [x-pack]
[2017-04-21T16:19:25,783][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
[2017-04-21T16:19:28,381][INFO ][o.e.n.Node               ] initialized
[2017-04-21T16:19:28,381][INFO ][o.e.n.Node               ] [4NynSsv] starting ...
[2017-04-21T16:19:28,653][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: 59:29:07:60:83:8c:94:96
[2017-04-21T16:19:28,730][INFO ][o.e.t.TransportService   ] [4NynSsv] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2017-04-21T16:19:31,826][INFO ][o.e.c.s.ClusterService   ] [4NynSsv] new_master {4NynSsv}{4NynSsv9RHS8EYIvevF1vQ}{_A8_cvJKQDirLQTHdntn7Q}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-04-21T16:19:31,857][INFO ][o.e.h.HttpServer         ] [4NynSsv] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2017-04-21T16:19:31,857][INFO ][o.e.n.Node               ] [4NynSsv] started
[2017-04-21T16:19:31,939][INFO ][o.e.g.GatewayService     ] [4NynSsv] recovered [0] indices into cluster_state
[2017-04-21T16:19:32,369][INFO ][o.e.l.LicenseService     ] [4NynSsv] license [e10d1dc3-f224-4806-bcc2-7dbef9219ddb] mode [trial] - valid
[2017-04-21T16:19:38,619][INFO ][o.e.c.m.MetaDataCreateIndexService] [4NynSsv] [.monitoring-es-2-2017.04.21] creating index, cause [auto(bulk api)], templates [.monitoring-es-2], shards [1]/[1], mappings [shards, _default_, node, index_stats, index_recovery, cluster_state, cluster_stats, indices_stats, node_stats]
[2017-04-21T16:19:38,733][INFO ][o.e.c.m.MetaDataCreateIndexService] [4NynSsv] [.monitoring-data-2] creating index, cause [auto(bulk api)], templates [.monitoring-data-2], shards [1]/[1], mappings [logstash, node, _default_, kibana, cluster_info]
[2017-04-21T16:19:38,983][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [cluster_stats]
[2017-04-21T16:19:39,037][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [node_stats]
[2017-04-21T16:19:49,185][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [index_stats]
[2017-04-21T16:19:49,266][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [cluster_stats]
[2017-04-21T16:19:49,319][INFO ][o.e.c.m.MetaDataMappingService] [4NynSsv] [.monitoring-es-2-2017.04.21/b2GEwCVlSiidTF_QcpAs2A] update_mapping [indices_stats]

You can expose multiple ports:

Any other thoughts?

Thanks in advance.


#4

I'm not sure if this is useful, but I tried curling the ports directly:

$ curl localhost:9200
{
  "name" : "4NynSsv",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "kOhN_OIrRoyAUc15sJEs7A",
  "version" : {
    "number" : "5.2.2",
    "build_hash" : "f9d9b74",
    "build_date" : "2017-02-24T17:26:45.835Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.1"
  },
  "tagline" : "You Know, for Search"
}
$ curl localhost:9300
curl: (52) Empty reply from server
$ curl localhost:9400
curl: (7) Failed to connect to localhost port 9400: Connection refused

I did the last one just to see if I would get a different reply for a invalid port.


(Simon Willnauer) #5

hmm we usually try to detect if you are trying to talk to a tranport port via HTTP so you should get a reply with a line like this: This is not a HTTP port do you get this if you do curl -XGET http://localhost:9300?


#6

I get the same error as before:

$ curl -XGET http://localhost:9300
curl: (52) Empty reply from server

(Simon Willnauer) #7

when I do this I get this on 5.3

curl -XGET http://localhost:9300
This is not a HTTP port

#8

So all I ran was the docker command:

docker run -e ES_JAVA_OPTS='-Xms1g -Xmx1g' --name elasticsearch -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "xpack.security.enabled=false" -d -p 9200:9200 -p 9300:9300 docker.elastic.co/elasticsearch/elasticsearch:5.2.2

and that curl. I'm on a Mac. Not sure if that makes a difference. I'm also using docker beta:

Version 17.05.0-ce-rc1-mac8 (16582)
Channel: edge
73d01bb48e

What are your specs?


#9

OK, if I do the curl INSIDE the docker container, I get the same error that you get.


(Simon Willnauer) #10

what if you set this to 0.0.0.0 as well?


#11

That fixed everything. Thanks!!!!!


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.