None of the configured nodes are available when GC took more than 30 sec

Hi,

I am using Elasticsearch 7.1.1 version. To connect with elasticsearch node I am using java transport client of 7.1.1 version.

When I perform continuously insert, fetch and delete records from the index. (Night run).I am getting below exception intermittently. There was no Elasticsearch crash observed during this time.

org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{Hhe6eS0YQGi4YbKLh0AnTA}{localhost}{127.0.0.1:9300}]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:386) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:382) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:676) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:45) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:52) ~[elasticsearch-7.1.1.jar:7.1.1]
at com.klera.index.provider.es.ElasticConnectionManager.getEsHealthStatus(ElasticConnectionManager.java:261) [datastoresdk-2.5.29.jar:?]
at com.klera.index.provider.es.ElasticSearchIndexCommunicator.getEsHealthStatus(ElasticSearchIndexCommunicator.java:206) [datastoresdk-2.5.29.jar:?]
at com.klera.serviceregistryservice.datastorage.management.ESCommunicator.getEsHealthStatus(ESCommunicator.java:1053) [serviceregistryservice-2.5.29.jar:?]
at com.klera.serviceregistryservice.indexrecycle.IndexReadOnlyManager.performTask(IndexReadOnlyManager.java:76) [serviceregistryservice-2.5.29.jar:?]
at com.klera.serviceregistryservice.management.MonitorThread.run(MonitorThread.java:54) [serviceregistryservice-2.5.29.jar:?]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_201]

Observation: Elastic was taking more than 30 sec for GC at that time.
Settings:
client.transport.ping_timeout : 90s
client.transport.nodes_sampler_interval : 5s
client.transport.sniff : false
cluster.name : mydemocluster

For this run I have single Elastic node.

I read some topics here and found that if Garbage collection time is more than ping timeout time than we can get NoNodeAvailableException.

In my case ping timeout is 90 sec and garbage collection time is 42 sec even though I am getting exception.

There is no connection issue as I am using client and server on same machine.

  1. Is my understanding is correct regarding 'If Garbage collection time is more than ping timeout time than we can get NoNodeAvailableException'?
  2. is something missing in my transport client setting?

Please suggest me so that I will not face NoNodeAvailableException during long GC.

Please revert me back if you required more information.

Thanks in advance.

Could you enable TRACE logging for org.elasticsearch.client.transport.TransportClientNodesService, wait until you get another NoNodeAvailableException, and then share the resulting logs here?

Thank @DavidTurner.

I will update the logs.

Is my understanding is correct regarding 'If Garbage collection time is more than ping timeout time than we can get NoNodeAvailableException'?

That is correct, but that is not the only explanation for a NoNodeAvailableException.

Please find logs as follow

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#6]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{ufxph0mOTFqNd77APNGWQw}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#2]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{8z0rKL0SQe6E5gs9TihwVA}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

@DavidTurner Provided logs when NoNodeAvailableException occured. at that time GC logs was not present.

ES Logs for your refernce.
[2020-02-14T09:02:29,508][INFO ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][58306] overhead, spent [401ms] collecting in the last [1s]
[2020-02-14T09:06:00,924][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][young][58515][4705] duration [2.1s], collections [1]/[3.1s], total [2.1s]/[8.5m], memory [1.1gb]->[746.9mb]/[1.9gb], all_pools {[young] [465mb]->[4.7mb]/[532.5mb]}{[survivor] [47.9mb]->[25.8mb]/[66.5mb]}{[old] [679.1mb]->[716.5mb]/[1.3gb]}
[2020-02-14T09:06:00,926][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][58515] overhead, spent [2.1s] collecting in the last [3.1s]
[2020-02-14T09:17:29,127][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][young][59201][4762] duration [1.5s], collections [1]/[2.4s], total [1.5s]/[8.5m], memory [1.1gb]->[840.3mb]/[1.9gb], all_pools {[young] [307.2mb]->[1018.8kb]/[532.5mb]}{[survivor] [50mb]->[42.2mb]/[66.5mb]}{[old] [774.9mb]->[797.1mb]/[1.3gb]}
[2020-02-14T09:17:29,128][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][59201] overhead, spent [1.5s] collecting in the last [2.4s]
[2020-02-14T09:22:26,404][INFO ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][59498] overhead, spent [276ms] collecting in the last [1s]
[2020-02-14T09:56:57,018][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][young][61566][4908] duration [1s], collections [1]/[1.3s], total [1s]/[8.6m], memory [1.4gb]->[1gb]/[1.9gb], all_pools {[young] [411.9mb]->[4.9mb]/[532.5mb]}{[survivor] [48.4mb]->[25.8mb]/[66.5mb]}{[old] [1017mb]->[1gb]/[1.3gb]}
[2020-02-14T09:56:57,020][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][61566] overhead, spent [1s] collecting in the last [1.3s]

Could you edit your posts to format these logs properly using the </> button, and check the preview pane before posting? They're very hard to read as they are now.

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#6]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{ufxph0mOTFqNd77APNGWQw}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#2]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{8z0rKL0SQe6E5gs9TihwVA}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

Thanks, that's much better.

Permission denied indicates that this client is being forbidden from connecting to 127.0.0.1:9300 by something in your environment. Note that this connection is not being rejected by Elasticsearch itself; it is not even talking to Elasticsearch at this point, so this error is coming from something in the environment between this client and Elasticsearch.

Thanks @DavidTurner for your help. I will look our environment.