# (ES 0.90.1) Cannot connect to elasticsearch cluster after a node is removed

**URL:** <https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343>\
**Category:** Elasticsearch\
**Created:** [March 13, 2014, 5:54am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343 "2014-03-13T05:54:07Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![deer](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@deer](https://discuss.elastic.co/u/deer)\
**Post date:** [March 13, 2014, 5:54am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/1 "2014-03-13T05:54:07Z")

</div>

Hi All,

This is the log for the case.

The node 10.1.4.196 is removed at 14:08 due to machine reboot, the client keeps trying to connect to the elasticsearch cluster but fails.  
Master Node :  
[2014-03-08 14:08:26,531][INFO][cluster.service] [10.1.4.197:9202] removed {[10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]],},  
reason: zen-disco-node\_failed([10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout

Client :  
2014-03-08 14:15:36,184 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0x50dc218f]], closing connection  
java.net.NoRouteToHostException: No route to host

(The cluster health at this moment is Yellow and there is no unassigned shard.)

The node is back at 14:25, the client can successfully connected to the cluster again.  
Client :  
2014-03-08 14:25:20,597 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0xf24d85d7]], closing connection  
java.net.NoRouteToHostException: No route to host

Master Node :  
[2014-03-08 14:25:57,984][INFO][cluster.service] [10.1.4.197:9202] added {[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]],}, reason: zen-disco-receive(join from node[[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]]])

(The cluster health at this moment is Green.)

In the above case, the client should be able to connect to the cluster even a node is removed from the cluster.

For the client, the connection is created as followings :  
Settings settings = ImmutableSettings.settingsBuilder()  
.put("cluster.name", "clustername")  
.put("client.transport.sniff", true)  
.build();  
TransportClient client = new TransportClient(settings);

```
    client.addTransportAddress(new InetSocketTransportAddress(
            "10.1.4.195" /* hostname */, 9300 /* port */));

    client.addTransportAddress(new InetSocketTransportAddress(

```

"10.1.4.196" /\* hostname _/, 9300 /_ port _/));  
client.addTransportAddress(new InetSocketTransportAddress(  
"10.1.4.197" /_ hostname _/, 9300 /_ port \*/));

The master node is 10.1.4.197 while the node being removed is 10.1.4.196.

For the cluster setting, all setting is using the default except the the discovery.zen.minimum\_master\_nodes  
which is set to 3.

Is there any problem for the above setting which cause this issue?

Thanks.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 13, 2014, 6:03am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/2 "2014-03-13T06:03:44Z")

</div>

It looks like a networking issue, at least based on "No route to host" in  
the error.  
Can you ping the master when this is happening, what about doing a telnet  
test?

Regards,  
Mark Walkom

Infrastructure Engineer  
Campaign Monitor  
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)

On 13 March 2014 16:54, Hui [dannyhui1103@gmail.com](mailto:dannyhui1103@gmail.com) wrote:

> Hi All,
> 
> This is the log for the case.
> 
> The node 10.1.4.196 is removed at 14:08 due to machine reboot, the client keeps trying to connect to the elasticsearch cluster but fails.  
> Master Node :  
> [2014-03-08 14:08:26,531][INFO][cluster.service] [10.1.4.197:9202] removed {[10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]],},  
> reason: zen-disco-node\_failed([10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
> 
> Client :  
> 2014-03-08 14:15:36,184 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0x50dc218f]], closing connection  
> java.net.NoRouteToHostException: No route to host
> 
> (The cluster health at this moment is Yellow and there is no unassigned shard.)
> 
> The node is back at 14:25, the client can successfully connected to the cluster again.  
> Client :  
> 2014-03-08 14:25:20,597 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0xf24d85d7]], closing connection  
> java.net.NoRouteToHostException: No route to host
> 
> Master Node :  
> [2014-03-08 14:25:57,984][INFO][cluster.service] [10.1.4.197:9202] added {[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]],}, reason: zen-disco-receive(join from node[[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]]])
> 
> (The cluster health at this moment is Green.)
> 
> In the above case, the client should be able to connect to the cluster even a node is removed from the cluster.
> 
> For the client, the connection is created as followings :
> 
> ```
> Settings settings = ImmutableSettings.settingsBuilder()
> .put("cluster.name", "clustername")
> .put("client.transport.sniff", true)
> 
> .build();
> 
> TransportClient client = new TransportClient(settings);
> 
> client.addTransportAddress(new InetSocketTransportAddress(
> "10.1.4.195" /* hostname */, 9300 /* port */));
> 
> client.addTransportAddress(new InetSocketTransportAddress(
> 
> ```
> 
> "10.1.4.196" /\* hostname _/, 9300 /_ port _/));  
> client.addTransportAddress(new InetSocketTransportAddress(  
> "10.1.4.197" /_ hostname _/, 9300 /_ port \*/));
> 
> The master node is 10.1.4.197 while the node being removed is 10.1.4.196.
> 
> For the cluster setting, all setting is using the default except the the discovery.zen.minimum\_master\_nodes  
> which is set to 3.
> 
> Is there any problem for the above setting which cause this issue?
> 
> Thanks.
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ahrjC1y7iDzQpb4UvrOnDbMRYnW%3D-u78TTBGLSwBc3Ow%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ahrjC1y7iDzQpb4UvrOnDbMRYnW%3D-u78TTBGLSwBc3Ow%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![deer](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@deer](https://discuss.elastic.co/u/deer)\
**Post date:** [March 13, 2014, 6:10am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/3 "2014-03-13T06:10:22Z")

</div>

Hi Mark,

Thanks for replying.

The master (10.1.4.197) and other nodes can be reached while the problem  
node(10.1.4.196) is not reachable.  
So, we can see the cluster status at that moment

"status" : "yellow",  
"timed\_out" : false,  
"unassigned\_shards" : 0,

On Thursday, March 13, 2014 2:03:44 PM UTC+8, Mark Walkom wrote:

> It looks like a networking issue, at least based on "No route to host" in  
> the error.  
> Can you ping the master when this is happening, what about doing a telnet  
> test?
> 
> Regards,  
> Mark Walkom
> 
> Infrastructure Engineer  
> Campaign Monitor  
> email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \<javascript:\>  
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> 
> On 13 March 2014 16:54, Hui \<[dannyh...@gmail.com](mailto:dannyh...@gmail.com) \<javascript:\>\> wrote:
> 
> > Hi All,
> > 
> > This is the log for the case.
> > 
> > The node 10.1.4.196 is removed at 14:08 due to machine reboot, the client keeps trying to connect to the elasticsearch cluster but fails.
> > 
> > Master Node :  
> > [2014-03-08 14:08:26,531][INFO][cluster.service] [10.1.4.197:9202] removed {[10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]],},  
> > reason: zen-disco-node\_failed([10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
> > 
> > Client :  
> > 2014-03-08 14:15:36,184 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0x50dc218f]], closing connection  
> > java.net.NoRouteToHostException: No route to host
> > 
> > (The cluster health at this moment is Yellow and there is no unassigned shard.)
> > 
> > The node is back at 14:25, the client can successfully connected to the cluster again.
> > 
> > Client :
> > 
> > 2014-03-08 14:25:20,597 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0xf24d85d7]], closing connection  
> > java.net.NoRouteToHostException: No route to host
> > 
> > Master Node :
> > 
> > [2014-03-08 14:25:57,984][INFO][cluster.service] [10.1.4.197:9202] added {[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]],}, reason: zen-disco-receive(join from node[[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]]])
> > 
> > (The cluster health at this moment is Green.)
> > 
> > In the above case, the client should be able to connect to the cluster even a node is removed from the cluster.
> > 
> > For the client, the connection is created as followings :
> > 
> > ```
> > Settings settings = ImmutableSettings.settingsBuilder()
> > .put("cluster.name", "clustername")
> > 
> > .put("client.transport.sniff", true)
> > 
> > .build();
> >     
> > 
> > TransportClient client = new TransportClient(settings);
> > 
> > client.addTransportAddress(new InetSocketTransportAddress(
> > "10.1.4.195" /* hostname */, 9300 /* port */));
> > 
> > client.addTransportAddress(new InetSocketTransportAddress(
> > 
> > ```
> > 
> > "10.1.4.196" /\* hostname _/, 9300 /_ port _/));  
> > client.addTransportAddress(new InetSocketTransportAddress(  
> > "10.1.4.197" /_ hostname _/, 9300 /_ port \*/));
> > 
> > The master node is 10.1.4.197 while the node being removed is 10.1.4.196.
> > 
> > For the cluster setting, all setting is using the default except the the discovery.zen.minimum\_master\_nodes  
> > which is set to 3.
> > 
> > Is there any problem for the above setting which cause this issue?
> > 
> > Thanks.
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fe322bdb-2726-4979-80d1-bb2f7372f28f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fe322bdb-2726-4979-80d1-bb2f7372f28f%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Dome\_C\_Wei](https://avatars.discourse-cdn.com/v4/letter/d/db5fbb/32.png) [@Dome\_C\_Wei](https://discuss.elastic.co/u/Dome_C_Wei)\
**Post date:** [March 13, 2014, 8:48am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/4 "2014-03-13T08:48:17Z")

</div>

That must be the service not open.

在 2014年3月13日星期四UTC+8下午2时10分22秒，Hui写道：

> Hi Mark,
> 
> Thanks for replying.
> 
> The master (10.1.4.197) and other nodes can be reached while the problem  
> node(10.1.4.196) is not reachable.  
> So, we can see the cluster status at that moment
> 
> "status" : "yellow",  
> "timed\_out" : false,  
> "unassigned\_shards" : 0,
> 
> On Thursday, March 13, 2014 2:03:44 PM UTC+8, Mark Walkom wrote:
> 
> > It looks like a networking issue, at least based on "No route to host" in  
> > the error.  
> > Can you ping the master when this is happening, what about doing a telnet  
> > test?
> > 
> > Regards,  
> > Mark Walkom
> > 
> > Infrastructure Engineer  
> > Campaign Monitor  
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)  
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > 
> > On 13 March 2014 16:54, Hui [dannyh...@gmail.com](mailto:dannyh...@gmail.com) wrote:
> > 
> > > Hi All,
> > > 
> > > This is the log for the case.
> > > 
> > > The node 10.1.4.196 is removed at 14:08 due to machine reboot, the client keeps trying to connect to the elasticsearch cluster but fails.
> > > 
> > > Master Node :  
> > > [2014-03-08 14:08:26,531][INFO][cluster.service] [10.1.4.197:9202] removed {[10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]],},  
> > > reason: zen-disco-node\_failed([10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
> > > 
> > > Client :  
> > > 2014-03-08 14:15:36,184 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0x50dc218f]], closing connection  
> > > java.net.NoRouteToHostException: No route to host
> > > 
> > > (The cluster health at this moment is Yellow and there is no unassigned shard.)
> > > 
> > > The node is back at 14:25, the client can successfully connected to the cluster again.
> > > 
> > > Client :
> > > 
> > > 2014-03-08 14:25:20,597 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0xf24d85d7]], closing connection  
> > > java.net.NoRouteToHostException: No route to host
> > > 
> > > Master Node :
> > > 
> > > [2014-03-08 14:25:57,984][INFO][cluster.service] [10.1.4.197:9202] added {[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]],}, reason: zen-disco-receive(join from node[[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]]])
> > > 
> > > (The cluster health at this moment is Green.)
> > > 
> > > In the above case, the client should be able to connect to the cluster even a node is removed from the cluster.
> > > 
> > > For the client, the connection is created as followings :
> > > 
> > > ```
> > > Settings settings = ImmutableSettings.settingsBuilder()
> > > .put("cluster.name", "clustername")
> > > 
> > > .put("client.transport.sniff", true)
> > > 
> > > .build();
> > >     
> > > 
> > > TransportClient client = new TransportClient(settings);
> > > 
> > > client.addTransportAddress(new InetSocketTransportAddress(
> > > "10.1.4.195" /* hostname */, 9300 /* port */));
> > > 
> > > client.addTransportAddress(new InetSocketTransportAddress(
> > > 
> > > ```
> > > 
> > > "10.1.4.196" /\* hostname _/, 9300 /_ port _/));  
> > > client.addTransportAddress(new InetSocketTransportAddress(  
> > > "10.1.4.197" /_ hostname _/, 9300 /_ port \*/));
> > > 
> > > The master node is 10.1.4.197 while the node being removed is 10.1.4.196.
> > > 
> > > For the cluster setting, all setting is using the default except the the discovery.zen.minimum\_master\_nodes  
> > > which is set to 3.
> > > 
> > > Is there any problem for the above setting which cause this issue?
> > > 
> > > Thanks.
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a24efac4-f61d-4aa9-913c-bf11eba2735f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a24efac4-f61d-4aa9-913c-bf11eba2735f%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![deer](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@deer](https://discuss.elastic.co/u/deer)\
**Post date:** [March 13, 2014, 9:15am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/5 "2014-03-13T09:15:36Z")

</div>

Hi Dome,

Do you mean the service of 10.1.4.196 is not open? Yes, the service should  
be stopped when it was rebooted.

But the master node 10.1.4.197 has removed the problem node 10.1.4.196 when  
it cannot ping the machine 10.1.4.196.

The cluster should be fine after this operation. Do I understand it wrongly?

Thanks

On Thursday, March 13, 2014 4:48:17 PM UTC+8, Dome.C.Wei wrote:

> That must be the service not open.
> 
> 在 2014年3月13日星期四UTC+8下午2时10分22秒，Hui写道：
> 
> > Hi Mark,
> > 
> > Thanks for replying.
> > 
> > The master (10.1.4.197) and other nodes can be reached while the problem  
> > node(10.1.4.196) is not reachable.  
> > So, we can see the cluster status at that moment
> > 
> > "status" : "yellow",  
> > "timed\_out" : false,  
> > "unassigned\_shards" : 0,
> > 
> > On Thursday, March 13, 2014 2:03:44 PM UTC+8, Mark Walkom wrote:
> > 
> > > It looks like a networking issue, at least based on "No route to host"  
> > > in the error.  
> > > Can you ping the master when this is happening, what about doing a  
> > > telnet test?
> > > 
> > > Regards,  
> > > Mark Walkom
> > > 
> > > Infrastructure Engineer  
> > > Campaign Monitor  
> > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)  
> > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > > 
> > > On 13 March 2014 16:54, Hui [dannyh...@gmail.com](mailto:dannyh...@gmail.com) wrote:
> > > 
> > > > Hi All,
> > > > 
> > > > This is the log for the case.
> > > > 
> > > > The node 10.1.4.196 is removed at 14:08 due to machine reboot, the client keeps trying to connect to the elasticsearch cluster but fails.
> > > > 
> > > > Master Node :  
> > > > [2014-03-08 14:08:26,531][INFO][cluster.service] [10.1.4.197:9202] removed {[10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]],},  
> > > > reason: zen-disco-node\_failed([10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
> > > > 
> > > > Client :  
> > > > 2014-03-08 14:15:36,184 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0x50dc218f]], closing connection  
> > > > java.net.NoRouteToHostException: No route to host
> > > > 
> > > > (The cluster health at this moment is Yellow and there is no unassigned shard.)
> > > > 
> > > > The node is back at 14:25, the client can successfully connected to the cluster again.
> > > > 
> > > > Client :
> > > > 
> > > > 2014-03-08 14:25:20,597 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0xf24d85d7]], closing connection  
> > > > java.net.NoRouteToHostException: No route to host
> > > > 
> > > > Master Node :
> > > > 
> > > > [2014-03-08 14:25:57,984][INFO][cluster.service] [10.1.4.197:9202] added {[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]],}, reason: zen-disco-receive(join from node[[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]]])
> > > > 
> > > > (The cluster health at this moment is Green.)
> > > > 
> > > > In the above case, the client should be able to connect to the cluster even a node is removed from the cluster.
> > > > 
> > > > For the client, the connection is created as followings :
> > > > 
> > > > ```
> > > > Settings settings = ImmutableSettings.settingsBuilder()
> > > > .put("cluster.name", "clustername")
> > > > 
> > > > .put("client.transport.sniff", true)
> > > > 
> > > > .build();
> > > >     
> > > > 
> > > > TransportClient client = new TransportClient(settings);
> > > > 
> > > > client.addTransportAddress(new InetSocketTransportAddress(
> > > > "10.1.4.195" /* hostname */, 9300 /* port */));
> > > > 
> > > > client.addTransportAddress(new InetSocketTransportAddress(
> > > > 
> > > > ```
> > > > 
> > > > "10.1.4.196" /\* hostname _/, 9300 /_ port _/));  
> > > > client.addTransportAddress(new InetSocketTransportAddress(  
> > > > "10.1.4.197" /_ hostname _/, 9300 /_ port \*/));
> > > > 
> > > > The master node is 10.1.4.197 while the node being removed is  
> > > > 10.1.4.196.
> > > > 
> > > > For the cluster setting, all setting is using the default except the  
> > > > the discovery.zen.minimum\_master\_nodes which is set to 3.
> > > > 
> > > > Is there any problem for the above setting which cause this issue?
> > > > 
> > > > Thanks.
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4f6ec6ac-ac8b-4a09-b338-2d8c6e225777%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4f6ec6ac-ac8b-4a09-b338-2d8c6e225777%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![echin1999\_2](https://avatars.discourse-cdn.com/v4/letter/e/f4b2a3/32.png) [@echin1999\_2](https://discuss.elastic.co/u/echin1999_2)\
**Post date:** [March 13, 2014, 2:48pm UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/6 "2014-03-13T14:48:07Z")

</div>

Hi.  
that would be my assumption as well. By the way, I am getting this same  
warning that you are getting. Very similar scenario (2 nodes in a cluster

- all works fine when everything is running. Warning appears on client if  
one of the nodes is taken down).

I am using v. 0.90 - not sure if that matters.

On Thursday, March 13, 2014 5:15:36 AM UTC-4, Hui wrote:

> Hi Dome,
> 
> Do you mean the service of 10.1.4.196 is not open? Yes, the service should  
> be stopped when it was rebooted.
> 
> But the master node 10.1.4.197 has removed the problem node 10.1.4.196  
> when it cannot ping the machine 10.1.4.196.
> 
> The cluster should be fine after this operation. Do I understand it  
> wrongly?
> 
> Thanks
> 
> On Thursday, March 13, 2014 4:48:17 PM UTC+8, Dome.C.Wei wrote:
> 
> > That must be the service not open.
> > 
> > 在 2014年3月13日星期四UTC+8下午2时10分22秒，Hui写道：
> > 
> > > Hi Mark,
> > > 
> > > Thanks for replying.
> > > 
> > > The master (10.1.4.197) and other nodes can be reached while the problem  
> > > node(10.1.4.196) is not reachable.  
> > > So, we can see the cluster status at that moment
> > > 
> > > "status" : "yellow",  
> > > "timed\_out" : false,  
> > > "unassigned\_shards" : 0,
> > > 
> > > On Thursday, March 13, 2014 2:03:44 PM UTC+8, Mark Walkom wrote:
> > > 
> > > > It looks like a networking issue, at least based on "No route to host"  
> > > > in the error.  
> > > > Can you ping the master when this is happening, what about doing a  
> > > > telnet test?
> > > > 
> > > > Regards,  
> > > > Mark Walkom
> > > > 
> > > > Infrastructure Engineer  
> > > > Campaign Monitor  
> > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)  
> > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > > > 
> > > > On 13 March 2014 16:54, Hui [dannyh...@gmail.com](mailto:dannyh...@gmail.com) wrote:
> > > > 
> > > > > Hi All,
> > > > > 
> > > > > This is the log for the case.
> > > > > 
> > > > > The node 10.1.4.196 is removed at 14:08 due to machine reboot, the client keeps trying to connect to the elasticsearch cluster but fails.
> > > > > 
> > > > > Master Node :  
> > > > > [2014-03-08 14:08:26,531][INFO][cluster.service] [10.1.4.197:9202] removed {[10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]],},  
> > > > > reason: zen-disco-node\_failed([10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
> > > > > 
> > > > > Client :  
> > > > > 2014-03-08 14:15:36,184 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0x50dc218f]], closing connection  
> > > > > java.net.NoRouteToHostException: No route to host
> > > > > 
> > > > > (The cluster health at this moment is Yellow and there is no unassigned shard.)
> > > > > 
> > > > > The node is back at 14:25, the client can successfully connected to the cluster again.
> > > > > 
> > > > > Client :
> > > > > 
> > > > > 2014-03-08 14:25:20,597 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0xf24d85d7]], closing connection  
> > > > > java.net.NoRouteToHostException: No route to host
> > > > > 
> > > > > Master Node :
> > > > > 
> > > > > [2014-03-08 14:25:57,984][INFO][cluster.service] [10.1.4.197:9202] added {[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]],}, reason: zen-disco-receive(join from node[[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]]])
> > > > > 
> > > > > (The cluster health at this moment is Green.)
> > > > > 
> > > > > In the above case, the client should be able to connect to the cluster even a node is removed from the cluster.
> > > > > 
> > > > > For the client, the connection is created as followings :
> > > > > 
> > > > > ```
> > > > > Settings settings = ImmutableSettings.settingsBuilder()
> > > > > .put("cluster.name", "clustername")
> > > > > 
> > > > > .put("client.transport.sniff", true)
> > > > > 
> > > > > .build();
> > > > >     
> > > > > 
> > > > > TransportClient client = new TransportClient(settings);
> > > > > 
> > > > > client.addTransportAddress(new InetSocketTransportAddress(
> > > > > "10.1.4.195" /* hostname */, 9300 /* port */));
> > > > > 
> > > > > client.addTransportAddress(new InetSocketTransportAddress(
> > > > > 
> > > > > ```
> > > > > 
> > > > > "10.1.4.196" /\* hostname _/, 9300 /_ port _/));  
> > > > > client.addTransportAddress(new InetSocketTransportAddress(  
> > > > > "10.1.4.197" /_ hostname _/, 9300 /_ port \*/));
> > > > > 
> > > > > The master node is 10.1.4.197 while the node being removed is  
> > > > > 10.1.4.196.
> > > > > 
> > > > > For the cluster setting, all setting is using the default except the  
> > > > > the discovery.zen.minimum\_master\_nodes which is set to 3.
> > > > > 
> > > > > Is there any problem for the above setting which cause this issue?
> > > > > 
> > > > > Thanks.
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > .  
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7485d1d2-9f1b-4fdd-bea8-d15d1b542904%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7485d1d2-9f1b-4fdd-bea8-d15d1b542904%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![echin1999\_2](https://avatars.discourse-cdn.com/v4/letter/e/f4b2a3/32.png) [@echin1999\_2](https://discuss.elastic.co/u/echin1999_2)\
**Post date:** [March 13, 2014, 3:54pm UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/7 "2014-03-13T15:54:53Z")

</div>

One more thing - I notice that functionally, the client is still able to  
communicate to the remaining active node. so I guess this warning is just  
a "warning". must be some background thread that periodically looks for  
the missing node, while the main Client instance can still communicate to  
the active node. would you be able to verify if its merely a warning for  
you? if so, i might just not worry about this for now.

On Thursday, March 13, 2014 5:15:36 AM UTC-4, Hui wrote:

> Hi Dome,
> 
> Do you mean the service of 10.1.4.196 is not open? Yes, the service should  
> be stopped when it was rebooted.
> 
> But the master node 10.1.4.197 has removed the problem node 10.1.4.196  
> when it cannot ping the machine 10.1.4.196.
> 
> The cluster should be fine after this operation. Do I understand it  
> wrongly?
> 
> Thanks
> 
> On Thursday, March 13, 2014 4:48:17 PM UTC+8, Dome.C.Wei wrote:
> 
> > That must be the service not open.
> > 
> > 在 2014年3月13日星期四UTC+8下午2时10分22秒，Hui写道：
> > 
> > > Hi Mark,
> > > 
> > > Thanks for replying.
> > > 
> > > The master (10.1.4.197) and other nodes can be reached while the problem  
> > > node(10.1.4.196) is not reachable.  
> > > So, we can see the cluster status at that moment
> > > 
> > > "status" : "yellow",  
> > > "timed\_out" : false,  
> > > "unassigned\_shards" : 0,
> > > 
> > > On Thursday, March 13, 2014 2:03:44 PM UTC+8, Mark Walkom wrote:
> > > 
> > > > It looks like a networking issue, at least based on "No route to host"  
> > > > in the error.  
> > > > Can you ping the master when this is happening, what about doing a  
> > > > telnet test?
> > > > 
> > > > Regards,  
> > > > Mark Walkom
> > > > 
> > > > Infrastructure Engineer  
> > > > Campaign Monitor  
> > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)  
> > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > > > 
> > > > On 13 March 2014 16:54, Hui [dannyh...@gmail.com](mailto:dannyh...@gmail.com) wrote:
> > > > 
> > > > > Hi All,
> > > > > 
> > > > > This is the log for the case.
> > > > > 
> > > > > The node 10.1.4.196 is removed at 14:08 due to machine reboot, the client keeps trying to connect to the elasticsearch cluster but fails.
> > > > > 
> > > > > Master Node :  
> > > > > [2014-03-08 14:08:26,531][INFO][cluster.service] [10.1.4.197:9202] removed {[10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]],},  
> > > > > reason: zen-disco-node\_failed([10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
> > > > > 
> > > > > Client :  
> > > > > 2014-03-08 14:15:36,184 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0x50dc218f]], closing connection  
> > > > > java.net.NoRouteToHostException: No route to host
> > > > > 
> > > > > (The cluster health at this moment is Yellow and there is no unassigned shard.)
> > > > > 
> > > > > The node is back at 14:25, the client can successfully connected to the cluster again.
> > > > > 
> > > > > Client :
> > > > > 
> > > > > 2014-03-08 14:25:20,597 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0xf24d85d7]], closing connection  
> > > > > java.net.NoRouteToHostException: No route to host
> > > > > 
> > > > > Master Node :
> > > > > 
> > > > > [2014-03-08 14:25:57,984][INFO][cluster.service] [10.1.4.197:9202] added {[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]],}, reason: zen-disco-receive(join from node[[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]]])
> > > > > 
> > > > > (The cluster health at this moment is Green.)
> > > > > 
> > > > > In the above case, the client should be able to connect to the cluster even a node is removed from the cluster.
> > > > > 
> > > > > For the client, the connection is created as followings :
> > > > > 
> > > > > ```
> > > > > Settings settings = ImmutableSettings.settingsBuilder()
> > > > > .put("cluster.name", "clustername")
> > > > > 
> > > > > .put("client.transport.sniff", true)
> > > > > 
> > > > > .build();
> > > > >     
> > > > > 
> > > > > TransportClient client = new TransportClient(settings);
> > > > > 
> > > > > client.addTransportAddress(new InetSocketTransportAddress(
> > > > > "10.1.4.195" /* hostname */, 9300 /* port */));
> > > > > 
> > > > > client.addTransportAddress(new InetSocketTransportAddress(
> > > > > 
> > > > > ```
> > > > > 
> > > > > "10.1.4.196" /\* hostname _/, 9300 /_ port _/));  
> > > > > client.addTransportAddress(new InetSocketTransportAddress(  
> > > > > "10.1.4.197" /_ hostname _/, 9300 /_ port \*/));
> > > > > 
> > > > > The master node is 10.1.4.197 while the node being removed is  
> > > > > 10.1.4.196.
> > > > > 
> > > > > For the cluster setting, all setting is using the default except the  
> > > > > the discovery.zen.minimum\_master\_nodes which is set to 3.
> > > > > 
> > > > > Is there any problem for the above setting which cause this issue?
> > > > > 
> > > > > Thanks.
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > .  
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a7d6c2d7-3f07-44cc-a15f-dc040c885508%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a7d6c2d7-3f07-44cc-a15f-dc040c885508%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![deer](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@deer](https://discuss.elastic.co/u/deer)\
**Post date:** [March 14, 2014, 3:21am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/8 "2014-03-14T03:21:53Z")

</div>

Hi Echin,

Since the problem node ip is defined in the client es connection by JAVA  
API, I guess the client will still try to connect to this node. So, there  
are such warnings.

It should be fine for client to keep working with the cluster. However, in  
my case, the java client is not reachable and timeout(through HTTP  
protocol).

I will try to create a testing cluster with same settings to test does the  
client work fine in this condition.

Thanks.

On Thursday, March 13, 2014 11:54:53 PM UTC+8, echin1999 wrote:

> One more thing - I notice that functionally, the client is still able to  
> communicate to the remaining active node. so I guess this warning is just  
> a "warning". must be some background thread that periodically looks for  
> the missing node, while the main Client instance can still communicate to  
> the active node. would you be able to verify if its merely a warning for  
> you? if so, i might just not worry about this for now.
> 
> On Thursday, March 13, 2014 5:15:36 AM UTC-4, Hui wrote:
> 
> > Hi Dome,
> > 
> > Do you mean the service of 10.1.4.196 is not open? Yes, the service  
> > should be stopped when it was rebooted.
> > 
> > But the master node 10.1.4.197 has removed the problem node 10.1.4.196  
> > when it cannot ping the machine 10.1.4.196.
> > 
> > The cluster should be fine after this operation. Do I understand it  
> > wrongly?
> > 
> > Thanks
> > 
> > On Thursday, March 13, 2014 4:48:17 PM UTC+8, Dome.C.Wei wrote:
> > 
> > > That must be the service not open.
> > > 
> > > 在 2014年3月13日星期四UTC+8下午2时10分22秒，Hui写道：
> > > 
> > > > Hi Mark,
> > > > 
> > > > Thanks for replying.
> > > > 
> > > > The master (10.1.4.197) and other nodes can be reached while the  
> > > > problem node(10.1.4.196) is not reachable.  
> > > > So, we can see the cluster status at that moment
> > > > 
> > > > "status" : "yellow",  
> > > > "timed\_out" : false,  
> > > > "unassigned\_shards" : 0,
> > > > 
> > > > On Thursday, March 13, 2014 2:03:44 PM UTC+8, Mark Walkom wrote:
> > > > 
> > > > > It looks like a networking issue, at least based on "No route to host"  
> > > > > in the error.  
> > > > > Can you ping the master when this is happening, what about doing a  
> > > > > telnet test?
> > > > > 
> > > > > Regards,  
> > > > > Mark Walkom
> > > > > 
> > > > > Infrastructure Engineer  
> > > > > Campaign Monitor  
> > > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)  
> > > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > > > > 
> > > > > On 13 March 2014 16:54, Hui [dannyh...@gmail.com](mailto:dannyh...@gmail.com) wrote:
> > > > > 
> > > > > > Hi All,
> > > > > > 
> > > > > > This is the log for the case.
> > > > > > 
> > > > > > The node 10.1.4.196 is removed at 14:08 due to machine reboot, the client keeps trying to connect to the elasticsearch cluster but fails.
> > > > > > 
> > > > > > Master Node :  
> > > > > > [2014-03-08 14:08:26,531][INFO][cluster.service] [10.1.4.197:9202] removed {[10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]],},  
> > > > > > reason: zen-disco-node\_failed([10.1.4.196:9202][\_sJrum34QWGqEkv8CvAtow][inet[/10.1.4.196:9302]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
> > > > > > 
> > > > > > Client :  
> > > > > > 2014-03-08 14:15:36,184 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0x50dc218f]], closing connection  
> > > > > > java.net.NoRouteToHostException: No route to host
> > > > > > 
> > > > > > (The cluster health at this moment is Yellow and there is no unassigned shard.)
> > > > > > 
> > > > > > The node is back at 14:25, the client can successfully connected to the cluster again.
> > > > > > 
> > > > > > Client :
> > > > > > 
> > > > > > 2014-03-08 14:25:20,597 WARN org.elasticsearch.transport.netty - [Bulldozer] exception caught on transport layer [[id: 0xf24d85d7]], closing connection  
> > > > > > java.net.NoRouteToHostException: No route to host
> > > > > > 
> > > > > > Master Node :
> > > > > > 
> > > > > > [2014-03-08 14:25:57,984][INFO][cluster.service] [10.1.4.197:9202] added {[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]],}, reason: zen-disco-receive(join from node[[10.1.4.196:9202][rFZ7k7XSSY231EgPoDfmFw][inet[/10.1.4.196:9302]]])
> > > > > > 
> > > > > > (The cluster health at this moment is Green.)
> > > > > > 
> > > > > > In the above case, the client should be able to connect to the cluster even a node is removed from the cluster.
> > > > > > 
> > > > > > For the client, the connection is created as followings :
> > > > > > 
> > > > > > ```
> > > > > > Settings settings = ImmutableSettings.settingsBuilder()
> > > > > > .put("cluster.name", "clustername")
> > > > > > 
> > > > > > .put("client.transport.sniff", true)
> > > > > > 
> > > > > > .build();
> > > > > >     
> > > > > > 
> > > > > > TransportClient client = new TransportClient(settings);
> > > > > > 
> > > > > > client.addTransportAddress(new InetSocketTransportAddress(
> > > > > > "10.1.4.195" /* hostname */, 9300 /* port */));
> > > > > > 
> > > > > > client.addTransportAddress(new InetSocketTransportAddress(
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > "10.1.4.196" /\* hostname _/, 9300 /_ port _/));  
> > > > > > client.addTransportAddress(new InetSocketTransportAddress(  
> > > > > > "10.1.4.197" /_ hostname _/, 9300 /_ port \*/));
> > > > > > 
> > > > > > The master node is 10.1.4.197 while the node being removed is  
> > > > > > 10.1.4.196.
> > > > > > 
> > > > > > For the cluster setting, all setting is using the default except the  
> > > > > > the discovery.zen.minimum\_master\_nodes which is set to 3.
> > > > > > 
> > > > > > Is there any problem for the above setting which cause this issue?
> > > > > > 
> > > > > > Thanks.
> > > > > > 
> > > > > > --  
> > > > > > You received this message because you are subscribed to the Google  
> > > > > > Groups "elasticsearch" group.  
> > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > To view this discussion on the web visit  
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b1f3adf5-723b-49aa-bffe-674c5ce930e5%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > .  
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e8973948-06fb-4675-a257-b28bbb09242d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e8973948-06fb-4675-a257-b28bbb09242d%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![deer](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@deer](https://discuss.elastic.co/u/deer)\
**Post date:** [March 14, 2014, 4:02am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/9 "2014-03-14T04:02:36Z")

</div>

Hi All,

After testing in another cluster, I found that the cluster can be connected  
but it was very slow.

At this moment, every normal request(~50ms) becomes 41732ms to 85984ms  
while the cluster is in Yellow health and there is no unassigned shard(s).

It becomes 50ms again after the problem node re-joins.

There is no exception log in the master node.

Thanks

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/840c93bb-f62a-4a65-9220-c725918436f8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/840c93bb-f62a-4a65-9220-c725918436f8%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![deer](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@deer](https://discuss.elastic.co/u/deer)\
**Post date:** [March 14, 2014, 10:16am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/10 "2014-03-14T10:16:13Z")

</div>

Sorry All.

I've verified that the slow problem is related to the  
reason failed to ping, tried [3] times, each with maximum [30s] timeout

Thanks.

On Friday, March 14, 2014 12:02:36 PM UTC+8, Hui wrote:

> Hi All,
> 
> After testing in another cluster, I found that the cluster can be  
> connected but it was very slow.
> 
> At this moment, every normal request(~50ms) becomes 41732ms to 85984ms  
> while the cluster is in Yellow health and there is no unassigned shard(s).
> 
> It becomes 50ms again after the problem node re-joins.
> 
> There is no exception log in the master node.
> 
> Thanks

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5f5a7739-c2b5-43cf-b040-346002d7281f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5f5a7739-c2b5-43cf-b040-346002d7281f%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:43am UTC](https://discuss.elastic.co/t/es-0-90-1-cannot-connect-to-elasticsearch-cluster-after-a-node-is-removed/16343/11 "2017-07-06T01:43:12Z")

</div>