# Node disconnects during merges on master?

**URL:** <https://discuss.elastic.co/t/node-disconnects-during-merges-on-master/8184>\
**Category:** Elasticsearch\
**Created:** [June 21, 2012, 4:48pm UTC](https://discuss.elastic.co/t/node-disconnects-during-merges-on-master/8184 "2012-06-21T16:48:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tim\_J](https://avatars.discourse-cdn.com/v4/letter/t/f17d59/32.png) [@Tim\_J](https://discuss.elastic.co/u/Tim_J)\
**Post date:** [June 21, 2012, 4:48pm UTC](https://discuss.elastic.co/t/node-disconnects-during-merges-on-master/8184/1 "2012-06-21T16:48:59Z")

</div>

Hey folks,  
I've recently been having trouble with a 20 node ES cluster where every  
other night one or more nodes will drop out of the cluster due to failed  
pings. Typically it's one or two nodes, always the same one or two, and  
roughly at the same time of night. At time these nodes are dropped the  
master seems to be doing a lot of merges and, presumably related, it has  
very high disk IO (close to 100%). The nodes that are dropped also tend to  
be busy with IO but not nearly as much as the master.

It's also worth noting that the master logs show the debug output "using  
[concurrent] merge scheduler with max\_thread\_count[3]" on the nights when a  
problem occurs at roughly the time of the failures. It's not clear to me  
what that message indicates or why it chooses to display the message then  
since presumably the scheduler has been responsible for any prior merges  
that day. Is it getting reset? Is it starting some kind of larger merge?  
Seems very coincidental.

So, any thoughts on what might be going wrong or how to address it? I  
could just extended the failure detection ping timeouts but that seems like  
it's hiding the symptom without tracking down the cause.

Thanks,  
-Tim

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:23am UTC](https://discuss.elastic.co/t/node-disconnects-during-merges-on-master/8184/2 "2017-07-06T03:23:04Z")

</div>

Closed on July 6, 2017, 3:23am UTC