# Transport client frequent "node disconnected" messages **URL:** \ **Category:** Elasticsearch\ **Created:** [April 24, 2019, 3:02pm UTC](https://discuss.elastic.co/t/transport-client-frequent-node-disconnected-messages/178281 "2019-04-24T15:02:22Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Chris\_Gatihi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_gatihi/32/25820_2.png) [@Chris\_Gatihi](https://discuss.elastic.co/u/Chris_Gatihi)\ **Post date:** [April 24, 2019, 3:02pm UTC](https://discuss.elastic.co/t/transport-client-frequent-node-disconnected-messages/178281/1 "2019-04-24T15:02:23Z")
Hi, We are on ES 5.5 and use the Transport Client with sniff set to true to connect our services to our ES cluster via a load balancer. We only need the load balancer to discover the nodes (they change dynamically) but after we establish a connection to them (which we keep open and don't close) we don't want to connect to the load balancer. As far as I understand, addresses associated with the Transport Client via `TransportClient#addTransportAddress(TransportAddress)` are only used for _initial connection_ after which the transport will discover the data nodes to maintain connection to. But we are seeing lots of these messages in our logs: `org.elasticsearch.transport.ConnectTransportException: [][192.168.192.9:9300] connect_timeout[30s]` Where the IP specified is for the load balancer. Why would we be seeing these messages? Do we need to use the `TransportClient#removeTransportAddress(TransportAddress)` functionality? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 24, 2019, 3:33pm UTC](https://discuss.elastic.co/t/transport-client-frequent-node-disconnected-messages/178281/2 "2019-04-24T15:33:21Z")
> [@Chris\_Gatihi](#): > > Why would we be seeing these messages? The short answer is that this node is timing out trying to connect to your load balancer, which suggests there might be some problem with the load balancer. Are you seeing these messages even when the load balancer is healthy? I think that the client will expect to be able to connect to all the addresses you give it via `addTransportAddress` on an ongoing basis. Although these addresses are mostly used for the initial connections, they may also be important if the client needs to reconnect to the cluster. ---
**Author:** ![Chris\_Gatihi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_gatihi/32/25820_2.png) [@Chris\_Gatihi](https://discuss.elastic.co/u/Chris_Gatihi)\ **Post date:** [April 24, 2019, 6:40pm UTC](https://discuss.elastic.co/t/transport-client-frequent-node-disconnected-messages/178281/3 "2019-04-24T18:40:53Z")
> [@DavidTurner](#): > > Are you seeing these messages even when the load balancer is healthy? Thanks for your reply @DavidTurner. These are happening consistently for us (almost every minute), so either that means the load balancer is never healthy or that they are happening even when the load balancer is healthy. We are passing a third parameter to the method `PreBuiltXPackTransportClient(Settings settings, Collection> plugins, HostFailureListener hostFailureListener)` when instantiating our Transport Client connection. We are seeing this listener getting called when the node disconnects after establishing a connection. I assume this would also get called if the initial connection fails to be established? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 25, 2019, 9:35am UTC](https://discuss.elastic.co/t/transport-client-frequent-node-disconnected-messages/178281/4 "2019-04-25T09:35:41Z")
Ah, sorry, I misread the exception message. The `connect_timeout[30s]` message is always included, even if the connection attempt didn't time out. Could you share the whole exception, including any stack traces and any inner exceptions (and their stack traces and so on). ---
**Author:** ![Chris\_Gatihi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_gatihi/32/25820_2.png) [@Chris\_Gatihi](https://discuss.elastic.co/u/Chris_Gatihi)\ **Post date:** [April 25, 2019, 1:34pm UTC](https://discuss.elastic.co/t/transport-client-frequent-node-disconnected-messages/178281/5 "2019-04-25T13:34:32Z")
> [@DavidTurner](#): > > The `connect_timeout[30s]` message is always included, even if the connection attempt didn't time out. Oh, interesting. That seems like it could be misleading. Here's all I see for the stacktrace: ``` org.elasticsearch.transport.ConnectTransportException: [][192.168.192.9:9300] connect_timeout[30s] at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:361) at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:548) at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:116) at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:351) at org.elasticsearch.client.transport.TransportClientNodesService$SniffNodesSampler$1.doRun(TransportClientNodesService.java:506) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: internal-FGHW3GK20XPB-360902321.eu-central-1.elb.amazonaws.com/192.168.192.9:9300 at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 common frames omitted ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 25, 2019, 1:47pm UTC](https://discuss.elastic.co/t/transport-client-frequent-node-disconnected-messages/178281/6 "2019-04-25T13:47:14Z")
> [@Chris\_Gatihi](#): > > ```plaintext > Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: internal-FGHW3GK20XPB-360902321.eu-central-1.elb.amazonaws.com/192.168.192.9:9300 > > ``` Definitely a timeout ☹ I'd be tempted to grab the traffic with `tcpdump` and look for connections that aren't being opened properly. That would give us a definite answer about whether we should look harder at the load balancer or at the Elasticsearch side. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 23, 2019, 1:47pm UTC](https://discuss.elastic.co/t/transport-client-frequent-node-disconnected-messages/178281/7 "2019-05-23T13:47:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.