Elasticsearch Transport Client fails to recovery connection after cluster restart

merlob · May 9, 2017, 8:49am

Hi,

I'm using Elasticsearch 5.4.0 with Transport Client with the following problems:

you must launch the application several times before it can connect to the cluster.
when connection is established, after a cluster restart, the connection is no more recovered, with this stack:

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{s-HKw1m4S9aMgCkx5iBuYg}{192.168.203.128}{192.168.203.128:9500}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:348)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:246)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:366)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:69)

The Transport Client is setting as following:

            Settings.Builder settingsBuilder = Settings.builder();

		settingsBuilder.put("cluster.name", "globevmes5");
		settingsBuilder.put("client.transport.sniff", true);
		 	
		client = new PreBuiltTransportClient(settingsBuilder.build());
		try {
			client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("192.168.203.128"), 9500));
			
		} catch (Exception e) {
			System.out.println(e.getMessage());	
		}

and Elasticsearch tcp port is 9500.

With previous elasticsearch 5.3.2 it worked fine.

Any ideas to solve it?

dadoonet · May 9, 2017, 9:13am

Can you share the logs of the elasticsearch node?

merlob · May 9, 2017, 9:20am

[2017-05-09T11:14:24,336][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1077556979/jna7634687564598757394.tmp: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/jna--1077556979/jna7634687564598757394.tmp)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_102]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_102]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_102]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_102]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_102]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0.jar:4.4.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0.jar:4.4.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.4.0.jar:4.4.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_102]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_102]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0]
[2017-05-09T11:14:24,342][WARN ][o.e.b.Natives            ] cannot check if running as root because JNA is not available
[2017-05-09T11:14:24,342][WARN ][o.e.b.Natives            ] cannot register console handler because JNA is not available
[2017-05-09T11:14:24,344][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2017-05-09T11:14:24,344][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2017-05-09T11:14:24,493][INFO ][o.e.n.Node               ] [globevmes5-node] initializing ...
[2017-05-09T11:14:24,615][INFO ][o.e.e.NodeEnvironment    ] [globevmes5-node] using [1] data paths, mounts [[/methode (/dev/mapper/VolGroup01-LogVol02)]], net usable_space [8.1gb], net total_space [72.8gb], spins? [possibly], types [ext3]
[2017-05-09T11:14:24,615][INFO ][o.e.e.NodeEnvironment    ] [globevmes5-node] heap size [1007.3mb], compressed ordinary object pointers [true]
[2017-05-09T11:14:24,659][INFO ][o.e.n.Node               ] [globevmes5-node] node name [globevmes5-node], node ID [M1_iHcSKRX6wHkD_Va0uDg]
[2017-05-09T11:14:24,659][INFO ][o.e.n.Node               ] [globevmes5-node] version[5.4.0], pid[31204], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/2.6.18-194.el5/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_102/25.102-b14]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [aggs-matrix-stats]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [ingest-common]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [lang-expression]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [lang-groovy]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [lang-mustache]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [lang-painless]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [percolator]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [reindex]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [transport-netty3]
[2017-05-09T11:14:26,981][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded module [transport-netty4]
[2017-05-09T11:14:26,982][INFO ][o.e.p.PluginsService     ] [globevmes5-node] loaded plugin [eom-elasticsearch-plugin]
[2017-05-09T11:14:28,406][INFO ][o.e.d.DiscoveryModule    ] [globevmes5-node] using discovery type [zen]
[2017-05-09T11:14:28,972][INFO ][o.e.n.Node               ] [globevmes5-node] initialized
[2017-05-09T11:14:28,972][INFO ][o.e.n.Node               ] [globevmes5-node] starting ...
[2017-05-09T11:14:29,100][INFO ][o.e.t.TransportService   ] [globevmes5-node] publish_address {192.168.203.128:9500}, bound_addresses {192.168.203.128:9500}
[2017-05-09T11:14:29,106][INFO ][o.e.b.BootstrapChecks    ] [globevmes5-node] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-05-09T11:14:32,227][INFO ][o.e.c.s.ClusterService   ] [globevmes5-node] new_master {globevmes5-node}{M1_iHcSKRX6wHkD_Va0uDg}{j8GR_IGFSfW2y502jd-SKA}{192.168.203.128}{192.168.203.128:9500}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-05-09T11:14:32,379][INFO ][o.e.h.n.Netty4HttpServerTransport] [globevmes5-node] publish_address {192.168.203.128:9400}, bound_addresses {192.168.203.128:9400}
[2017-05-09T11:14:32,386][INFO ][o.e.n.Node               ] [globevmes5-node] started
[2017-05-09T11:14:32,590][INFO ][o.e.g.GatewayService     ] [globevmes5-node] recovered [6] indices into cluster_state
[2017-05-09T11:14:33,173][INFO ][o.e.c.r.a.AllocationService] [globevmes5-node] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
[2017-05-09T11:15:02,357][INFO ][o.e.c.r.a.DiskThresholdMonitor] [globevmes5-node] low disk watermark [85%] exceeded on [M1_iHcSKRX6wHkD_Va0uDg][globevmes5-node][/methode/meth01/mnt/elasticsearch-5.4.0/data/nodes/0] free: 8.1gb[11.1%], replicas will not be assigned to this node

Setting "client.transport.sniff" to false works fine.

thx

dadoonet · May 9, 2017, 9:34am

May be open an issue then?

billnbell · May 16, 2017, 2:55pm

I tried client.transport.sniff true and false and get the same issue. Node not available. Works fine in 5.3.2 TransportClient.

List nodes = client.connectedNodes();

This is empty....

system · June 13, 2017, 2:55pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
TransportClient NoNodeAvailableException Elasticsearch	4	747	November 22, 2017
Transport Client Error both on 5.3.2 and 5.4.0 Elasticsearch	6	1404	June 26, 2017
TransportClient and NoNodeAvailableException Elasticsearch	6	593	July 6, 2017
NoNodeAvailableException problem Elasticsearch	6	1993	July 28, 2017
Recovering from NoNodeAvailableException Elasticsearch	5	3333	July 6, 2017

Elasticsearch Transport Client fails to recovery connection after cluster restart

Related topics