# Indefinite hang when adding nodes?

**URL:** <https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578>\
**Category:** Elasticsearch\
**Created:** [June 8, 2011, 11:02pm UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578 "2011-06-08T23:02:35Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Derek\_Wollenstein](https://avatars.discourse-cdn.com/v4/letter/d/e274bd/32.png) [@Derek\_Wollenstein](https://discuss.elastic.co/u/Derek_Wollenstein)\
**Post date:** [June 8, 2011, 11:02pm UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/1 "2011-06-08T23:02:35Z")

</div>

```
I'm not sure if I'm doing things wrong, but I seem to be

```

encountering some problems adding registered addresses to a  
TransportClient. The ScheduledConnectNodeSampler class inside of  
TransportClientNodesService.java would block indefinitely trying to  
refresh data from a given node. If that node happened to be dead at  
the time, the entire client would come to a halt. I've modified this  
to take a timeout in milliseconds to use when retrieving nodesInfo and  
gotten this to run more smoothly, but I don't know if this indicates  
that I'm doing something wrong. Thanks for your help

--Derek

---

<div class="post-metadata">

**Author:** ![Eric\_Mill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_mill/32/3124_2.png) [@Eric\_Mill](https://discuss.elastic.co/u/Eric_Mill)\
**Post date:** [June 9, 2011, 1:36am UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/2 "2011-06-09T01:36:07Z")

</div>

I wonder if this would explain the strange hanging behavior I've been seeing  
intermittently when trying to index documents. I can make it go away by  
blowing away the data repo and starting over, and it may not reoccur for a  
long time. The next time it happens, I'll start investigating the node  
endpoints to see if it's happening while a node is dead.

-- Eric

On Wed, Jun 8, 2011 at 7:02 PM, Derek Wollenstein [derek@klout.com](mailto:derek@klout.com) wrote:

> I'm not sure if I'm doing things wrong, but I seem to be  
> encountering some problems adding registered addresses to a  
> TransportClient. The ScheduledConnectNodeSampler class inside of  
> TransportClientNodesService.java would block indefinitely trying to  
> refresh data from a given node. If that node happened to be dead at  
> the time, the entire client would come to a halt. I've modified this  
> to take a timeout in milliseconds to use when retrieving nodesInfo and  
> gotten this to run more smoothly, but I don't know if this indicates  
> that I'm doing something wrong. Thanks for your help
> 
> --Derek

---

<div class="post-metadata">

**Author:** ![fashionalwallet](https://avatars.discourse-cdn.com/v4/letter/f/839c29/32.png) [@fashionalwallet](https://discuss.elastic.co/u/fashionalwallet)\
**Post date:** [June 9, 2011, 8:53am UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/3 "2011-06-09T08:53:27Z")

</div>

- deleted -

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 9, 2011, 7:09pm UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/4 "2011-06-09T19:09:51Z")

</div>

Its not really indefinite, I am not sure why it hangs, but eventually it will fail on closed socket. Or, if it tries to connect to a broken socket, under some OS, it will take time to identify that the socket is not there to connect to (by default, it has 30s timeout). Do you really see it as indefinite?

On Thursday, June 9, 2011 at 2:02 AM, Derek Wollenstein wrote:

> I'm not sure if I'm doing things wrong, but I seem to be  
> encountering some problems adding registered addresses to a  
> TransportClient. The ScheduledConnectNodeSampler class inside of  
> TransportClientNodesService.java would block indefinitely trying to  
> refresh data from a given node. If that node happened to be dead at  
> the time, the entire client would come to a halt. I've modified this  
> to take a timeout in milliseconds to use when retrieving nodesInfo and  
> gotten this to run more smoothly, but I don't know if this indicates  
> that I'm doing something wrong. Thanks for your help
> 
> --Derek

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 9, 2011, 7:10pm UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/5 "2011-06-09T19:10:23Z")

</div>

Did not full understand your problem...

On Thursday, June 9, 2011 at 4:36 AM, Eric Mill wrote:

> I wonder if this would explain the strange hanging behavior I've been seeing intermittently when trying to index documents. I can make it go away by blowing away the data repo and starting over, and it may not reoccur for a long time. The next time it happens, I'll start investigating the node endpoints to see if it's happening while a node is dead.
> 
> -- Eric
> 
> On Wed, Jun 8, 2011 at 7:02 PM, Derek Wollenstein \<[derek@klout.com](mailto:derek@klout.com) ([mailto:derek@klout.com](mailto:derek@klout.com))\> wrote:
> 
> > I'm not sure if I'm doing things wrong, but I seem to be  
> > encountering some problems adding registered addresses to a  
> > TransportClient. The ScheduledConnectNodeSampler class inside of  
> > TransportClientNodesService.java would block indefinitely trying to  
> > refresh data from a given node. If that node happened to be dead at  
> > the time, the entire client would come to a halt. I've modified this  
> > to take a timeout in milliseconds to use when retrieving nodesInfo and  
> > gotten this to run more smoothly, but I don't know if this indicates  
> > that I'm doing something wrong. Thanks for your help
> > 
> > --Derek

---

<div class="post-metadata">

**Author:** ![Derek\_Wollenstein](https://avatars.discourse-cdn.com/v4/letter/d/e274bd/32.png) [@Derek\_Wollenstein](https://discuss.elastic.co/u/Derek_Wollenstein)\
**Post date:** [June 9, 2011, 9:08pm UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/6 "2011-06-09T21:08:24Z")

</div>

Shay -

```
When I say "indefinite", what I mean is that I wrote some code

```

that does the following (without the rest of the class):  
Settings settings =  
ImmutableSettings.settingsBuilder().put("cluster.name", clusterName).  
.put(NetworkService.TcpSettings.TCP\_CONNECT\_TIMEOUT,  
1).build();  
TransportClient indexer = new TransportClient(settings);  
String hosts = new String {"host1","host2"/\* ,... \*/,  
"hostN"};

```
        for (int i = 0; i < hosts.length; i++) {
            LOG.info(" host : " + hosts[i]);
            indexer = indexer.addTransportAddress(new

```

InetSocketTransportAddress(hosts[i], 9300));  
}

If I run this I'll see the output  
host: host1  
host: host2  
[... pause for several minutes ...]  
And then I'll abort this with Ctrl+C. I've had this run fine  
depending on connectivity, but if there's any pause at all it seems to  
hang forever. I didn't get anything to improve without modifying the  
ScheduledConnectNodesSampler to have an internal timeout

On Jun 9, 12:10 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Did not full understand your problem...
> 
> On Thursday, June 9, 2011 at 4:36 AM, Eric Mill wrote:
> 
> > I wonder if this would explain the strange hanging behavior I've been seeing intermittently when trying to index documents. I can make it go away by blowing away the data repo and starting over, and it may not reoccur for a long time. The next time it happens, I'll start investigating the node endpoints to see if it's happening while a node is dead.
> 
> > -- Eric
> 
> > On Wed, Jun 8, 2011 at 7:02 PM, Derek Wollenstein \<[de...@klout.com](mailto:de...@klout.com) ([mailto:de...@klout.com](mailto:de...@klout.com))\> wrote:
> > 
> > > I'm not sure if I'm doing things wrong, but I seem to be  
> > > encountering some problems adding registered addresses to a  
> > > TransportClient. The ScheduledConnectNodeSampler class inside of  
> > > TransportClientNodesService.java would block indefinitely trying to  
> > > refresh data from a given node. If that node happened to be dead at  
> > > the time, the entire client would come to a halt. I've modified this  
> > > to take a timeout in milliseconds to use when retrieving nodesInfo and  
> > > gotten this to run more smoothly, but I don't know if this indicates  
> > > that I'm doing something wrong. Thanks for your help
> 
> > > --Derek

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 10, 2011, 12:17am UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/7 "2011-06-10T00:17:58Z")

</div>

When you set it to 1, it means 1 millisecond, can you try and set it to "1s"? Also, when you see this hang, can you issue a thread dump, lets see where its stuck (gist it please).

On Friday, June 10, 2011 at 12:08 AM, Derek Wollenstein wrote:

> Shay -
> 
> When I say "indefinite", what I mean is that I wrote some code  
> that does the following (without the rest of the class):  
> Settings settings =  
> ImmutableSettings.settingsBuilder().put("cluster.name ([http://cluster.name](http://cluster.name))", clusterName).  
> .put(NetworkService.TcpSettings.TCP\_CONNECT\_TIMEOUT,  
> 1).build();  
> TransportClient indexer = new TransportClient(settings);  
> String hosts = new String {"host1","host2"/\* ,... \*/,  
> "hostN"};
> 
> for (int i = 0; i \< hosts.length; i++) {  
> LOG.info ([http://LOG.info](http://LOG.info))(" host : " + hosts[i]);  
> indexer = indexer.addTransportAddress(new  
> InetSocketTransportAddress(hosts[i], 9300));  
> }
> 
> If I run this I'll see the output  
> host: host1  
> host: host2  
> [... pause for several minutes ...]  
> And then I'll abort this with Ctrl+C. I've had this run fine  
> depending on connectivity, but if there's any pause at all it seems to  
> hang forever. I didn't get anything to improve without modifying the  
> ScheduledConnectNodesSampler to have an internal timeout
> 
> On Jun 9, 12:10 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote:
> 
> > Did not full understand your problem...
> > 
> > On Thursday, June 9, 2011 at 4:36 AM, Eric Mill wrote:
> > 
> > > I wonder if this would explain the strange hanging behavior I've been seeing intermittently when trying to index documents. I can make it go away by blowing away the data repo and starting over, and it may not reoccur for a long time. The next time it happens, I'll start investigating the node endpoints to see if it's happening while a node is dead.
> > 
> > > -- Eric
> > 
> > > On Wed, Jun 8, 2011 at 7:02 PM, Derek Wollenstein \<[de...@klout.com](mailto:de...@klout.com) ([mailto:de...@klout.com](mailto:de...@klout.com) ([http://klout.com](http://klout.com)))\> wrote:
> > > 
> > > > I'm not sure if I'm doing things wrong, but I seem to be  
> > > > encountering some problems adding registered addresses to a  
> > > > TransportClient. The ScheduledConnectNodeSampler class inside of  
> > > > TransportClientNodesService.java would block indefinitely trying to  
> > > > refresh data from a given node. If that node happened to be dead at  
> > > > the time, the entire client would come to a halt. I've modified this  
> > > > to take a timeout in milliseconds to use when retrieving nodesInfo and  
> > > > gotten this to run more smoothly, but I don't know if this indicates  
> > > > that I'm doing something wrong. Thanks for your help
> > 
> > > > --Derek

---

<div class="post-metadata">

**Author:** ![Derek\_Wollenstein](https://avatars.discourse-cdn.com/v4/letter/d/e274bd/32.png) [@Derek\_Wollenstein](https://discuss.elastic.co/u/Derek_Wollenstein)\
**Post date:** [June 14, 2011, 10:23pm UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/8 "2011-06-14T22:23:02Z")

</div>

Shay-  
I just got this to reproduce itself using default settings. This  
basically is what happens when a previously connected node disappears  
when using a local gateway, and unicast discovery. I attempted to  
check for the same lock being waited on after running jstack,  
sleeping, and running jstack again.

I've put the apparently hung thread in

> <https://gist.github.com/anonymous/1026062>

Actually I'll go ahead and updated this to be the complete thread  
dump.

Threads "elasticsearch[cached]-pool-1-thread-80",  
"elasticsearch[cached]-pool-1-thread-79", "elasticsearch[cached]-  
pool-1-thread-78", and at least one more are exhibiting this problem.

On Jun 9, 5:17 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> When you set it to 1, it means 1 millisecond, can you try and set it to "1s"? Also, when you see this hang, can you issue a thread dump, lets see where its stuck (gist it please).
> 
> On Friday, June 10, 2011 at 12:08 AM, Derek Wollenstein wrote:
> 
> > Shay -
> 
> > When I say "indefinite", what I mean is that I wrote some code  
> > that does the following (without the rest of the class):  
> > Settings settings =  
> > ImmutableSettings.settingsBuilder().put("cluster.name ([http://cluster.name](http://cluster.name))", clusterName).  
> > .put(NetworkService.TcpSettings.TCP\_CONNECT\_TIMEOUT,  
> > 1).build();  
> > TransportClient indexer = new TransportClient(settings);  
> > String hosts = new String {"host1","host2"/\* ,... \*/,  
> > "hostN"};
> 
> > for (int i = 0; i \< hosts.length; i++) {  
> > LOG.info ([http://LOG.info](http://LOG.info))(" host : " + hosts[i]);  
> > indexer = indexer.addTransportAddress(new  
> > InetSocketTransportAddress(hosts[i], 9300));  
> > }
> 
> > If I run this I'll see the output  
> > host: host1  
> > host: host2  
> > [... pause for several minutes ...]  
> > And then I'll abort this with Ctrl+C. I've had this run fine  
> > depending on connectivity, but if there's any pause at all it seems to  
> > hang forever. I didn't get anything to improve without modifying the  
> > ScheduledConnectNodesSampler to have an internal timeout
> 
> > On Jun 9, 12:10 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote:
> > 
> > > Did not full understand your problem...
> 
> > > On Thursday, June 9, 2011 at 4:36 AM, Eric Mill wrote:
> > > 
> > > > I wonder if this would explain the strange hanging behavior I've been seeing intermittently when trying to index documents. I can make it go away by blowing away the data repo and starting over, and it may not reoccur for a long time. The next time it happens, I'll start investigating the node endpoints to see if it's happening while a node is dead.
> 
> > > > -- Eric
> 
> > > > On Wed, Jun 8, 2011 at 7:02 PM, Derek Wollenstein \<[de...@klout.com](mailto:de...@klout.com) ([mailto:de...@klout.com](mailto:de...@klout.com) ([http://klout.com](http://klout.com)))\> wrote:
> > > > 
> > > > > I'm not sure if I'm doing things wrong, but I seem to be  
> > > > > encountering some problems adding registered addresses to a  
> > > > > TransportClient. The ScheduledConnectNodeSampler class inside of  
> > > > > TransportClientNodesService.java would block indefinitely trying to  
> > > > > refresh data from a given node. If that node happened to be dead at  
> > > > > the time, the entire client would come to a halt. I've modified this  
> > > > > to take a timeout in milliseconds to use when retrieving nodesInfo and  
> > > > > gotten this to run more smoothly, but I don't know if this indicates  
> > > > > that I'm doing something wrong. Thanks for your help
> 
> > > > > --Derek

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 14, 2011, 10:54pm UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/9 "2011-06-14T22:54:02Z")

</div>

So, it stays hang on that lock (which basically waits for the response to come back)? Basically, when a node is identified as disconnected (on the client side), then it also "releases" all the ongoing messages that has been sent to it. How easy is it to recreate it? If it is simple, would you mind terribly jumping on IRC and I can create debug version of ES to test whats happening?

On Wednesday, June 15, 2011 at 1:23 AM, Derek Wollenstein wrote:

> Shay-  
> I just got this to reproduce itself using default settings. This  
> basically is what happens when a previously connected node disappears  
> when using a local gateway, and unicast discovery. I attempted to  
> check for the same lock being waited on after running jstack,  
> sleeping, and running jstack again.
> 
> I've put the apparently hung thread in  
> [Thread when ScheduledConnectNodeSampler.run is stuck · GitHub](https://gist.github.com/1026062)
> 
> Actually I'll go ahead and updated this to be the complete thread  
> dump.
> 
> Threads "elasticsearch[cached]-pool-1-thread-80",  
> "elasticsearch[cached]-pool-1-thread-79", "elasticsearch[cached]-  
> pool-1-thread-78", and at least one more are exhibiting this problem.
> 
> On Jun 9, 5:17 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote:
> 
> > When you set it to 1, it means 1 millisecond, can you try and set it to "1s"? Also, when you see this hang, can you issue a thread dump, lets see where its stuck (gist it please).
> > 
> > On Friday, June 10, 2011 at 12:08 AM, Derek Wollenstein wrote:
> > 
> > > Shay -
> > 
> > > When I say "indefinite", what I mean is that I wrote some code  
> > > that does the following (without the rest of the class):  
> > > Settings settings =  
> > > ImmutableSettings.settingsBuilder().put("cluster.name ([http://cluster.name](http://cluster.name))", clusterName).  
> > > .put(NetworkService.TcpSettings.TCP\_CONNECT\_TIMEOUT,  
> > > 1).build();  
> > > TransportClient indexer = new TransportClient(settings);  
> > > String hosts = new String {"host1","host2"/\* ,... \*/,  
> > > "hostN"};
> > 
> > > for (int i = 0; i \< hosts.length; i++) {  
> > > LOG.info ([http://LOG.info](http://LOG.info))(" host : " + hosts[i]);  
> > > indexer = indexer.addTransportAddress(new  
> > > InetSocketTransportAddress(hosts[i], 9300));  
> > > }
> > 
> > > If I run this I'll see the output  
> > > host: host1  
> > > host: host2  
> > > [... pause for several minutes ...]  
> > > And then I'll abort this with Ctrl+C. I've had this run fine  
> > > depending on connectivity, but if there's any pause at all it seems to  
> > > hang forever. I didn't get anything to improve without modifying the  
> > > ScheduledConnectNodesSampler to have an internal timeout
> > 
> > > On Jun 9, 12:10 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote:
> > > 
> > > > Did not full understand your problem...
> > 
> > > > On Thursday, June 9, 2011 at 4:36 AM, Eric Mill wrote:
> > > > 
> > > > > I wonder if this would explain the strange hanging behavior I've been seeing intermittently when trying to index documents. I can make it go away by blowing away the data repo and starting over, and it may not reoccur for a long time. The next time it happens, I'll start investigating the node endpoints to see if it's happening while a node is dead.
> > 
> > > > > -- Eric
> > 
> > > > > On Wed, Jun 8, 2011 at 7:02 PM, Derek Wollenstein \<[de...@klout.com](mailto:de...@klout.com) ([mailto:de...@klout.com](mailto:de...@klout.com) ([http://klout.com](http://klout.com)))\> wrote:
> > > > > 
> > > > > > I'm not sure if I'm doing things wrong, but I seem to be  
> > > > > > encountering some problems adding registered addresses to a  
> > > > > > TransportClient. The ScheduledConnectNodeSampler class inside of  
> > > > > > TransportClientNodesService.java would block indefinitely trying to  
> > > > > > refresh data from a given node. If that node happened to be dead at  
> > > > > > the time, the entire client would come to a halt. I've modified this  
> > > > > > to take a timeout in milliseconds to use when retrieving nodesInfo and  
> > > > > > gotten this to run more smoothly, but I don't know if this indicates  
> > > > > > that I'm doing something wrong. Thanks for your help
> > 
> > > > > > --Derek

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:03am UTC](https://discuss.elastic.co/t/indefinite-hang-when-adding-nodes/4578/10 "2017-07-06T04:03:34Z")

</div>