Failed to connect to node [..], removed from node list

Elastic Stack Elasticsearch
1

Hi,

I'm just testing elasticsearch clusters and how to handle node-downtimes in
the application.

I'm on version 0.18.6, two nodes on the same local machine.
Everything works fine tough, searching, filtering etc.

Shutting down one node (does not matter which), with strg+c causes the
application to log the following message:

2012-01-26 14:51:37: [org.elasticsearch.client.transport] - [Mother Night]
Failed to connect to node [#transport#-1][inet[/10.1.1.2:9300]], removed
from nodes list
org.elasticsearch.transport.ConnectTransportException:
[][inet[/10.1.1.2:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:560)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:503)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:482)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:124)
at
org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:262)
at
org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:245)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
... 3 more

In the first moment I thought "Fine, node is automatically removed from the
node list. Nothing to do for me".
The problem: The application logs this message every 5 Seconds.

Is there a method to remove the node temporary, so this message does not
appear so often?
How is the usually way to handle such failures?
Perhaps I want to shut the machine down forever, how do I do it, without
lots of errormessages?

Thx in advance
Greetings
Maik

2

You are using the TransportClient, and you added that node as an address to be connected to. So, it will try to re-establish the connection to that node. The logging is in DEBUG level, so you can reduce the logging level in your app.

On Thursday, January 26, 2012 at 4:01 PM, maik wrote:

Hi,

I'm just testing elasticsearch clusters and how to handle node-downtimes in the application.

I'm on version 0.18.6, two nodes on the same local machine.
Everything works fine tough, searching, filtering etc.

Shutting down one node (does not matter which), with strg+c causes the application to log the following message:

2012-01-26 14:51:37: [org.elasticsearch.client.transport] - [Mother Night] Failed to connect to node [#transport#-1][inet[/10.1.1.2:9300]], removed from nodes list
org.elasticsearch.transport.ConnectTransportException: [inet[/10.1.1.2:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:560)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:503)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:482)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:124)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:262)
at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:245)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
... 3 more

In the first moment I thought "Fine, node is automatically removed from the node list. Nothing to do for me".
The problem: The application logs this message every 5 Seconds.

Is there a method to remove the node temporary, so this message does not appear so often?
How is the usually way to handle such failures?
Perhaps I want to shut the machine down forever, how do I do it, without lots of errormessages?

Thx in advance
Greetings
Maik

3

Thx kimchy (was away for some days, so i could not thank you earlier :D)