None of the configured nodes are available when GC took more than 30 sec

Hi,

I am using Elasticsearch 7.1.1 version. To connect with elasticsearch node I am using java transport client of 7.1.1 version.

When I perform continuously insert, fetch and delete records from the index. (Night run).I am getting below exception intermittently. There was no Elasticsearch crash observed during this time.

org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{Hhe6eS0YQGi4YbKLh0AnTA}{localhost}{127.0.0.1:9300}]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:386) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:382) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:676) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:45) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:52) ~[elasticsearch-7.1.1.jar:7.1.1]
at com.klera.index.provider.es.ElasticConnectionManager.getEsHealthStatus(ElasticConnectionManager.java:261) [datastoresdk-2.5.29.jar:?]
at com.klera.index.provider.es.ElasticSearchIndexCommunicator.getEsHealthStatus(ElasticSearchIndexCommunicator.java:206) [datastoresdk-2.5.29.jar:?]
at com.klera.serviceregistryservice.datastorage.management.ESCommunicator.getEsHealthStatus(ESCommunicator.java:1053) [serviceregistryservice-2.5.29.jar:?]
at com.klera.serviceregistryservice.indexrecycle.IndexReadOnlyManager.performTask(IndexReadOnlyManager.java:76) [serviceregistryservice-2.5.29.jar:?]
at com.klera.serviceregistryservice.management.MonitorThread.run(MonitorThread.java:54) [serviceregistryservice-2.5.29.jar:?]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_201]

Observation: Elastic was taking more than 30 sec for GC at that time.
Settings:
client.transport.ping_timeout : 90s
client.transport.nodes_sampler_interval : 5s
client.transport.sniff : false
cluster.name : mydemocluster

For this run I have single Elastic node.

I read some topics here and found that if Garbage collection time is more than ping timeout time than we can get NoNodeAvailableException.

In my case ping timeout is 90 sec and garbage collection time is 42 sec even though I am getting exception.

There is no connection issue as I am using client and server on same machine.

  1. Is my understanding is correct regarding 'If Garbage collection time is more than ping timeout time than we can get NoNodeAvailableException'?
  2. is something missing in my transport client setting?

Please suggest me so that I will not face NoNodeAvailableException during long GC.

Please revert me back if you required more information.

Thanks in advance.

Could you enable TRACE logging for org.elasticsearch.client.transport.TransportClientNodesService, wait until you get another NoNodeAvailableException, and then share the resulting logs here?

Thank @DavidTurner.

I will update the logs.

Is my understanding is correct regarding 'If Garbage collection time is more than ping timeout time than we can get NoNodeAvailableException'?

That is correct, but that is not the only explanation for a NoNodeAvailableException.

Please find logs as follow

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#6]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{ufxph0mOTFqNd77APNGWQw}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#2]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{8z0rKL0SQe6E5gs9TihwVA}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

@DavidTurner Provided logs when NoNodeAvailableException occured. at that time GC logs was not present.

ES Logs for your refernce.
[2020-02-14T09:02:29,508][INFO ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][58306] overhead, spent [401ms] collecting in the last [1s]
[2020-02-14T09:06:00,924][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][young][58515][4705] duration [2.1s], collections [1]/[3.1s], total [2.1s]/[8.5m], memory [1.1gb]->[746.9mb]/[1.9gb], all_pools {[young] [465mb]->[4.7mb]/[532.5mb]}{[survivor] [47.9mb]->[25.8mb]/[66.5mb]}{[old] [679.1mb]->[716.5mb]/[1.3gb]}
[2020-02-14T09:06:00,926][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][58515] overhead, spent [2.1s] collecting in the last [3.1s]
[2020-02-14T09:17:29,127][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][young][59201][4762] duration [1.5s], collections [1]/[2.4s], total [1.5s]/[8.5m], memory [1.1gb]->[840.3mb]/[1.9gb], all_pools {[young] [307.2mb]->[1018.8kb]/[532.5mb]}{[survivor] [50mb]->[42.2mb]/[66.5mb]}{[old] [774.9mb]->[797.1mb]/[1.3gb]}
[2020-02-14T09:17:29,128][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][59201] overhead, spent [1.5s] collecting in the last [2.4s]
[2020-02-14T09:22:26,404][INFO ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][59498] overhead, spent [276ms] collecting in the last [1s]
[2020-02-14T09:56:57,018][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][young][61566][4908] duration [1s], collections [1]/[1.3s], total [1s]/[8.6m], memory [1.4gb]->[1gb]/[1.9gb], all_pools {[young] [411.9mb]->[4.9mb]/[532.5mb]}{[survivor] [48.4mb]->[25.8mb]/[66.5mb]}{[old] [1017mb]->[1gb]/[1.3gb]}
[2020-02-14T09:56:57,020][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][61566] overhead, spent [1s] collecting in the last [1.3s]

Could you edit your posts to format these logs properly using the </> button, and check the preview pane before posting? They're very hard to read as they are now.

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#6]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{ufxph0mOTFqNd77APNGWQw}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#2]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{8z0rKL0SQe6E5gs9TihwVA}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

Thanks, that's much better.

Permission denied indicates that this client is being forbidden from connecting to 127.0.0.1:9300 by something in your environment. Note that this connection is not being rejected by Elasticsearch itself; it is not even talking to Elasticsearch at this point, so this error is coming from something in the environment between this client and Elasticsearch.

Thanks @DavidTurner for your help. I will look our environment.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.