# ES client nor node can not join cluster while it performs maintenance or shard relocation

**URL:** <https://discuss.elastic.co/t/es-client-nor-node-can-not-join-cluster-while-it-performs-maintenance-or-shard-relocation/11236>\
**Category:** Elasticsearch\
**Created:** [March 21, 2013, 12:54pm UTC](https://discuss.elastic.co/t/es-client-nor-node-can-not-join-cluster-while-it-performs-maintenance-or-shard-relocation/11236 "2013-03-21T12:54:05Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Konstantins\_Trusins](https://avatars.discourse-cdn.com/v4/letter/k/91b2a8/32.png) [@Konstantins\_Trusins](https://discuss.elastic.co/u/Konstantins_Trusins)\
**Post date:** [March 21, 2013, 12:54pm UTC](https://discuss.elastic.co/t/es-client-nor-node-can-not-join-cluster-while-it-performs-maintenance-or-shard-relocation/11236/1 "2013-03-21T12:54:05Z")

</div>

Hi all

I have following situation. We use 20 node cluster with replication 1. We  
have 5 masters and minimal master count is set to 3. When cluster is  
initializing, relocating\_shards or creating new indice master node utilize  
1 CPU core to 100% and during that time other nodes can not join cluster  
reporting following error

[2013-03-20 10:50:20,815][DEBUG][discovery.zen] [srvd1573]  
using ping.timeout [3s], master\_election.filter\_client [true],  
master\_election.filter\_data [false]  
[2013-03-20 10:50:20,816][DEBUG][discovery.zen.elect] [srvd1573]  
using minimum\_master\_nodes [3]  
[2013-03-20 10:50:20,817][DEBUG][discovery.zen.fd] [srvd1573]  
[master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2013-03-20 10:50:20,822][DEBUG][discovery.zen.fd] [srvd1573]  
[node] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2013-03-20 10:50:21,115][DEBUG][gateway.local] [srvd1573]  
using initial\_shards [quorum], list\_timeout [30s]  
[2013-03-20 10:50:21,246][DEBUG][gateway.local.state.meta] [srvd1573]  
using gateway.local.auto\_import\_dangled [YES], with  
gateway.local.dangling\_timeout [2h]  
[2013-03-20 10:50:21,821][DEBUG][gateway.local.state.meta] [srvd1573] took  
575ms to load state  
[2013-03-20 10:50:23,207][DEBUG][gateway.local.state.shards] [srvd1573]  
took 1.3s to load started shards state  
[2013-03-20 10:50:23,211][INFO][node] [srvd1573]  
{0.20.4}[12314]: initialized  
[2013-03-20 10:50:23,211][INFO][node] [srvd1573]  
{0.20.4}[12314]: starting ...  
[2013-03-20 10:50:23,352][INFO][transport] [srvd1573]  
bound\_address {inet[/0.0.0.0:9300]}, publish\_address {inet[/x.x.x.x:9300]}  
[2013-03-20 10:50:26,381][DEBUG][discovery.zen] [srvd1573]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
[2013-03-20 10:50:29,394][DEBUG][discovery.zen] [srvd1573]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
[2013-03-20 10:50:32,405][DEBUG][discovery.zen] [srvd1573]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
[2013-03-20 10:50:35,416][DEBUG][discovery.zen] [srvd1573]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
[2013-03-20 10:50:38,424][DEBUG][discovery.zen] [srvd1573]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
[2013-03-20 10:50:41,431][DEBUG][discovery.zen] [srvd1573]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
[2013-03-20 10:50:44,437][DEBUG][discovery.zen] [srvd1573]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
[2013-03-20 10:50:47,444][DEBUG][discovery.zen] [srvd1573]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
[2013-03-20 10:50:50,483][DEBUG][discovery.zen] [srvd1573]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[srvk758][8jPTUZBKQr-OW3qRjNL3uA][inet[/x.x.x.x:9300]]{hosting=KV,  
max\_local\_storage\_nodes=1, master=true}], master [null]  
--\> target  
[[srvd1582][ZSLewNkzS8iEqn3q2FSyHw][inet[/x.x.x.x:9300]]{hosting=DL,  
max\_local\_storage\_nodes=1, master=true}], master [null]  
--\> target  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
[2013-03-20 10:50:51,205][DEBUG][discovery.zen.fd] [srvd1573]  
[master] starting fault detection against master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage  
\_nodes=1, master=true}], reason [initial\_join]  
[2013-03-20 10:50:52,235][DEBUG][discovery.zen.fd] [srvd1573]  
[master] pinging a master  
[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true} b  
ut we do not exists on it, act as if its master failure  
[2013-03-20 10:50:52,236][DEBUG][discovery.zen.fd] [srvd1573]  
[master] stopping fault detection against master  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage  
\_nodes=1, master=true}], reason [master failure, do not exists on master,  
act as master failure]  
[2013-03-20 10:50:52,236][INFO][discovery.zen] [srvd1573]  
master\_left  
[[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
hosting=M100, max\_local\_storage\_nodes=1, master=true}], reason [do n  
ot exists on master, act as master failure]  
[2013-03-20 10:50:52,238][INFO][discovery] [srvd1573]  
graylog2/icsr0HBQRZKhCsLp4QM8Nw  
[2013-03-20 10:50:52,256][INFO][http] [srvd1573]  
bound\_address {inet[/0.0.0.0:9200]}, publish\_address {inet[/x.x.x.x:9200]}  
[2013-03-20 10:50:52,256][INFO][node] [srvd1573]  
{0.20.4}[12314]: started

After new indice is created or shard relocation is done node can easily  
join cluster. How this problem can be solved?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [March 25, 2013, 3:56pm UTC](https://discuss.elastic.co/t/es-client-nor-node-can-not-join-cluster-while-it-performs-maintenance-or-shard-relocation/11236/2 "2013-03-25T15:56:10Z")

</div>

Could you run hot\_threads[http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-hot-threads.html](http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-hot-threads.html)on the master when it goes into this 100% CPU state and post the results  
here?

On Thursday, March 21, 2013 8:54:05 AM UTC-4, Konstantins Trusins wrote:

> Hi all
> 
> I have following situation. We use 20 node cluster with replication 1. We  
> have 5 masters and minimal master count is set to 3. When cluster is  
> initializing, relocating\_shards or creating new indice master node utilize  
> 1 CPU core to 100% and during that time other nodes can not join cluster  
> reporting following error
> 
> [2013-03-20 10:50:20,815][DEBUG][discovery.zen] [srvd1573]  
> using ping.timeout [3s], master\_election.filter\_client [true],  
> master\_election.filter\_data [false]  
> [2013-03-20 10:50:20,816][DEBUG][discovery.zen.elect] [srvd1573]  
> using minimum\_master\_nodes [3]  
> [2013-03-20 10:50:20,817][DEBUG][discovery.zen.fd] [srvd1573]  
> [master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
> [2013-03-20 10:50:20,822][DEBUG][discovery.zen.fd] [srvd1573]  
> [node] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
> [2013-03-20 10:50:21,115][DEBUG][gateway.local] [srvd1573]  
> using initial\_shards [quorum], list\_timeout [30s]  
> [2013-03-20 10:50:21,246][DEBUG][gateway.local.state.meta] [srvd1573]  
> using gateway.local.auto\_import\_dangled [YES], with  
> gateway.local.dangling\_timeout [2h]  
> [2013-03-20 10:50:21,821][DEBUG][gateway.local.state.meta] [srvd1573]  
> took 575ms to load state  
> [2013-03-20 10:50:23,207][DEBUG][gateway.local.state.shards] [srvd1573]  
> took 1.3s to load started shards state  
> [2013-03-20 10:50:23,211][INFO][node] [srvd1573]  
> {0.20.4}[12314]: initialized  
> [2013-03-20 10:50:23,211][INFO][node] [srvd1573]  
> {0.20.4}[12314]: starting ...  
> [2013-03-20 10:50:23,352][INFO][transport] [srvd1573]  
> bound\_address {inet[/0.0.0.0:9300]}, publish\_address {inet[/x.x.x.x:9300]}  
> [2013-03-20 10:50:26,381][DEBUG][discovery.zen] [srvd1573]  
> filtered ping responses: (filter\_client[true], filter\_data[false])  
> --\> target  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
> alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
> [2013-03-20 10:50:29,394][DEBUG][discovery.zen] [srvd1573]  
> filtered ping responses: (filter\_client[true], filter\_data[false])  
> --\> target  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
> alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
> [2013-03-20 10:50:32,405][DEBUG][discovery.zen] [srvd1573]  
> filtered ping responses: (filter\_client[true], filter\_data[false])  
> --\> target  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
> alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
> [2013-03-20 10:50:35,416][DEBUG][discovery.zen] [srvd1573]  
> filtered ping responses: (filter\_client[true], filter\_data[false])  
> --\> target  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
> alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
> [2013-03-20 10:50:38,424][DEBUG][discovery.zen] [srvd1573]  
> filtered ping responses: (filter\_client[true], filter\_data[false])  
> --\> target  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
> alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
> [2013-03-20 10:50:41,431][DEBUG][discovery.zen] [srvd1573]  
> filtered ping responses: (filter\_client[true], filter\_data[false])  
> --\> target  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
> alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
> [2013-03-20 10:50:44,437][DEBUG][discovery.zen] [srvd1573]  
> filtered ping responses: (filter\_client[true], filter\_data[false])  
> --\> target  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
> alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
> [2013-03-20 10:50:47,444][DEBUG][discovery.zen] [srvd1573]  
> filtered ping responses: (filter\_client[true], filter\_data[false])  
> --\> target  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
> alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
> [2013-03-20 10:50:50,483][DEBUG][discovery.zen] [srvd1573]  
> filtered ping responses: (filter\_client[true], filter\_data[false])  
> --\> target  
> [[srvk758][8jPTUZBKQr-OW3qRjNL3uA][inet[/x.x.x.x:9300]]{hosting=KV,  
> max\_local\_storage\_nodes=1, master=true}], master [null]  
> --\> target  
> [[srvd1582][ZSLewNkzS8iEqn3q2FSyHw][inet[/x.x.x.x:9300]]{hosting=DL,  
> max\_local\_storage\_nodes=1, master=true}], master [null]  
> --\> target  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=f  
> alse, hosting=M100, max\_local\_storage\_nodes=1, master=true}]  
> [2013-03-20 10:50:51,205][DEBUG][discovery.zen.fd] [srvd1573]  
> [master] starting fault detection against master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage  
> \_nodes=1, master=true}], reason [initial\_join]  
> [2013-03-20 10:50:52,235][DEBUG][discovery.zen.fd] [srvd1573]  
> [master] pinging a master  
> [srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true} b  
> ut we do not exists on it, act as if its master failure  
> [2013-03-20 10:50:52,236][DEBUG][discovery.zen.fd] [srvd1573]  
> [master] stopping fault detection against master  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage  
> \_nodes=1, master=true}], reason [master failure, do not exists on master,  
> act as master failure]  
> [2013-03-20 10:50:52,236][INFO][discovery.zen] [srvd1573]  
> master\_left  
> [[srvk768][0sAOH4RuQYWa3XraCuSOkg][inet[/x.x.x.x:9300]]{data=false,  
> hosting=M100, max\_local\_storage\_nodes=1, master=true}], reason [do n  
> ot exists on master, act as master failure]  
> [2013-03-20 10:50:52,238][INFO][discovery] [srvd1573]  
> graylog2/icsr0HBQRZKhCsLp4QM8Nw  
> [2013-03-20 10:50:52,256][INFO][http] [srvd1573]  
> bound\_address {inet[/0.0.0.0:9200]}, publish\_address {inet[/x.x.x.x:9200]}  
> [2013-03-20 10:50:52,256][INFO][node] [srvd1573]  
> {0.20.4}[12314]: started
> 
> After new indice is created or shard relocation is done node can easily  
> join cluster. How this problem can be solved?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:44am UTC](https://discuss.elastic.co/t/es-client-nor-node-can-not-join-cluster-while-it-performs-maintenance-or-shard-relocation/11236/3 "2017-07-06T02:44:44Z")

</div>