None of the configured nodes are available when GC took more than 30 sec

chelawatankit · February 12, 2020, 11:53am

Hi,

I am using Elasticsearch 7.1.1 version. To connect with elasticsearch node I am using java transport client of 7.1.1 version.

When I perform continuously insert, fetch and delete records from the index. (Night run).I am getting below exception intermittently. There was no Elasticsearch crash observed during this time.

org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{Hhe6eS0YQGi4YbKLh0AnTA}{localhost}{127.0.0.1:9300}]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:386) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:382) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:676) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:45) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:52) ~[elasticsearch-7.1.1.jar:7.1.1]
at com.klera.index.provider.es.ElasticConnectionManager.getEsHealthStatus(ElasticConnectionManager.java:261) [datastoresdk-2.5.29.jar:?]
at com.klera.index.provider.es.ElasticSearchIndexCommunicator.getEsHealthStatus(ElasticSearchIndexCommunicator.java:206) [datastoresdk-2.5.29.jar:?]
at com.klera.serviceregistryservice.datastorage.management.ESCommunicator.getEsHealthStatus(ESCommunicator.java:1053) [serviceregistryservice-2.5.29.jar:?]
at com.klera.serviceregistryservice.indexrecycle.IndexReadOnlyManager.performTask(IndexReadOnlyManager.java:76) [serviceregistryservice-2.5.29.jar:?]
at com.klera.serviceregistryservice.management.MonitorThread.run(MonitorThread.java:54) [serviceregistryservice-2.5.29.jar:?]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_201]

Observation: Elastic was taking more than 30 sec for GC at that time.
Settings:
client.transport.ping_timeout : 90s
client.transport.nodes_sampler_interval : 5s
client.transport.sniff : false
cluster.name : mydemocluster

For this run I have single Elastic node.

chelawatankit · February 12, 2020, 12:50pm

I read some topics here and found that if Garbage collection time is more than ping timeout time than we can get NoNodeAvailableException.

In my case ping timeout is 90 sec and garbage collection time is 42 sec even though I am getting exception.

There is no connection issue as I am using client and server on same machine.

Is my understanding is correct regarding 'If Garbage collection time is more than ping timeout time than we can get NoNodeAvailableException'?
is something missing in my transport client setting?

Please suggest me so that I will not face NoNodeAvailableException during long GC.

Please revert me back if you required more information.

Thanks in advance.

DavidTurner · February 12, 2020, 1:09pm

Could you enable TRACE logging for org.elasticsearch.client.transport.TransportClientNodesService, wait until you get another NoNodeAvailableException, and then share the resulting logs here?

chelawatankit · February 12, 2020, 1:22pm

Thank @DavidTurner.

I will update the logs.

Is my understanding is correct regarding 'If Garbage collection time is more than ping timeout time than we can get NoNodeAvailableException'?

DavidTurner · February 12, 2020, 1:46pm

That is correct, but that is not the only explanation for a NoNodeAvailableException.

chelawatankit · February 14, 2020, 9:54am

Please find logs as follow

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#6]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{ufxph0mOTFqNd77APNGWQw}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

chelawatankit · February 14, 2020, 9:54am

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#2]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{8z0rKL0SQe6E5gs9TihwVA}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

chelawatankit · February 14, 2020, 9:56am

@DavidTurner Provided logs when NoNodeAvailableException occured. at that time GC logs was not present.

ES Logs for your refernce.
[2020-02-14T09:02:29,508][INFO ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][58306] overhead, spent [401ms] collecting in the last [1s]
[2020-02-14T09:06:00,924][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][young][58515][4705] duration [2.1s], collections [1]/[3.1s], total [2.1s]/[8.5m], memory [1.1gb]->[746.9mb]/[1.9gb], all_pools {[young] [465mb]->[4.7mb]/[532.5mb]}{[survivor] [47.9mb]->[25.8mb]/[66.5mb]}{[old] [679.1mb]->[716.5mb]/[1.3gb]}
[2020-02-14T09:06:00,926][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][58515] overhead, spent [2.1s] collecting in the last [3.1s]
[2020-02-14T09:17:29,127][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][young][59201][4762] duration [1.5s], collections [1]/[2.4s], total [1.5s]/[8.5m], memory [1.1gb]->[840.3mb]/[1.9gb], all_pools {[young] [307.2mb]->[1018.8kb]/[532.5mb]}{[survivor] [50mb]->[42.2mb]/[66.5mb]}{[old] [774.9mb]->[797.1mb]/[1.3gb]}
[2020-02-14T09:17:29,128][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][59201] overhead, spent [1.5s] collecting in the last [2.4s]
[2020-02-14T09:22:26,404][INFO ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][59498] overhead, spent [276ms] collecting in the last [1s]
[2020-02-14T09:56:57,018][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][young][61566][4908] duration [1s], collections [1]/[1.3s], total [1s]/[8.6m], memory [1.4gb]->[1gb]/[1.9gb], all_pools {[young] [411.9mb]->[4.9mb]/[532.5mb]}{[survivor] [48.4mb]->[25.8mb]/[66.5mb]}{[old] [1017mb]->[1gb]/[1.3gb]}
[2020-02-14T09:56:57,020][WARN ][o.e.m.j.JvmGcMonitorService] [cinode_ct-i0073] [gc][61566] overhead, spent [1s] collecting in the last [1.3s]

DavidTurner · February 14, 2020, 9:59am

Could you edit your posts to format these logs properly using the </> button, and check the preview pane before posting? They're very hard to read as they are now.

chelawatankit · February 14, 2020, 10:49am

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#6]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{ufxph0mOTFqNd77APNGWQw}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

chelawatankit · February 14, 2020, 10:50am

2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#2]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{8z0rKL0SQe6E5gs9TihwVA}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more

DavidTurner · February 14, 2020, 11:33am

Thanks, that's much better.

chelawatankit:

Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
...
Caused by: java.net.SocketException: Permission denied: connect

Permission denied indicates that this client is being forbidden from connecting to 127.0.0.1:9300 by something in your environment. Note that this connection is not being rejected by Elasticsearch itself; it is not even talking to Elasticsearch at this point, so this error is coming from something in the environment between this client and Elasticsearch.

chelawatankit · February 14, 2020, 11:38am

Thanks @DavidTurner for your help. I will look our environment.

system · March 13, 2020, 11:38am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.