I am using Elasticsearch 7.1.1 version. To connect with elasticsearch node I am using java transport client of 7.1.1 version.
When I perform continuously insert, fetch and delete records from the index. (Night run).I am getting below exception intermittently. There was no Elasticsearch crash observed during this time.
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{Hhe6eS0YQGi4YbKLh0AnTA}{localhost}{127.0.0.1:9300}]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:386) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:382) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:676) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:45) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:52) ~[elasticsearch-7.1.1.jar:7.1.1]
at com.klera.index.provider.es.ElasticConnectionManager.getEsHealthStatus(ElasticConnectionManager.java:261) [datastoresdk-2.5.29.jar:?]
at com.klera.index.provider.es.ElasticSearchIndexCommunicator.getEsHealthStatus(ElasticSearchIndexCommunicator.java:206) [datastoresdk-2.5.29.jar:?]
at com.klera.serviceregistryservice.datastorage.management.ESCommunicator.getEsHealthStatus(ESCommunicator.java:1053) [serviceregistryservice-2.5.29.jar:?]
at com.klera.serviceregistryservice.indexrecycle.IndexReadOnlyManager.performTask(IndexReadOnlyManager.java:76) [serviceregistryservice-2.5.29.jar:?]
at com.klera.serviceregistryservice.management.MonitorThread.run(MonitorThread.java:54) [serviceregistryservice-2.5.29.jar:?]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_201]
Observation: Elastic was taking more than 30 sec for GC at that time.
Settings:
client.transport.ping_timeout : 90s
client.transport.nodes_sampler_interval : 5s
client.transport.sniff : false
cluster.name : mydemocluster
Could you enable TRACE logging for org.elasticsearch.client.transport.TransportClientNodesService, wait until you get another NoNodeAvailableException, and then share the resulting logs here?
2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#6]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{ufxph0mOTFqNd77APNGWQw}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#2]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{8z0rKL0SQe6E5gs9TihwVA}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Could you edit your posts to format these logs properly using the </> button, and check the preview pane before posting? They're very hard to read as they are now.
2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#6]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{ufxph0mOTFqNd77APNGWQw}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
2020-02-14 09:11:14,222 DEBUG [THREAD=elasticsearch[cinode_ct-i0073][generic][T#2]]-[TransportClientNodesService] [line 437] failed to connect to node [{#transport#-1}{8z0rKL0SQe6E5gs9TihwVA}{localhost}{127.0.0.1:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[transport-netty4-client-7.1.1.jar:7.1.1]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:290) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:201) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:263) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Permission denied: connect: localhost/127.0.0.1:9300
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Caused by: java.net.SocketException: Permission denied: connect
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_201]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
... 18 more
Permission denied indicates that this client is being forbidden from connecting to 127.0.0.1:9300 by something in your environment. Note that this connection is not being rejected by Elasticsearch itself; it is not even talking to Elasticsearch at this point, so this error is coming from something in the environment between this client and Elasticsearch.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.