# Runaway ES TransportClient threads after ES node failure
**URL:** https://discuss.elastic.co/t/runaway-es-transportclient-threads-after-es-node-failure/6083
**Category:** Elasticsearch
**Created:** [December 6, 2011, 10:34pm UTC](https://discuss.elastic.co/t/runaway-es-transportclient-threads-after-es-node-failure/6083 "2011-12-06T22:34:20Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [December 6, 2011, 10:34pm UTC](https://discuss.elastic.co/t/runaway-es-transportclient-threads-after-es-node-failure/6083/1 "2011-12-06T22:34:20Z")
I posted this on IRC, but obviously my GMT+11 Timezone is not friendly, so
as a backup I post the text here for anyone that might have experience in
this:
I have an application using a TransportClient configured to connect to a
2-node ES cluster (i'll leave aside for now why we have to use the
TransportClient, but it's rationale..)
one of the ES nodes hand a faulty backplane and died.
ES of course kept on trucking with the other node
however since that event the application client has burnt a hell of a lot
of CPU
which looking at the thread dumps look to be the "New I/O client worker
#1-5 daemon" style threads used by ES.
I thought somehow with the one ES node dead there's some looping logic
trying to re-establish connection to it.
so I waited till the Dell guys replaced the backplane and we restored that
node
once back in green state I was hoping the CPU burn would go away, but alas
no.
now looking at one of our other instances running in a similar config, I
note the ES app threads are always runnable because of the NIO, but they're
generally in a sleep state looking at them.
has anyone else seen this sort of problem?
I'm just gathering a known 'good' thread dump to compare this with.
Here's a gist: [https://gist.github.com/1440329](https://gist.github.com/1440329)
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [December 7, 2011, 3:27pm UTC](https://discuss.elastic.co/t/runaway-es-transportclient-threads-after-es-node-failure/6083/2 "2011-12-07T15:27:27Z")
Heya Paul, which version are you using? It sounds like a problem in netty
(the networking layer elasticsearch uses) which have been fixed in the
latest version ([Potential infinite loop in NIO client socket boss thread · Issue #74 · netty/netty · GitHub](https://github.com/netty/netty/issues/74)) and included in
the latest version of elasticsearch (0.18.5).
On Wed, Dec 7, 2011 at 12:34 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
> I posted this on IRC, but obviously my GMT+11 Timezone is not friendly, so
> as a backup I post the text here for anyone that might have experience in
> this:
>
> I have an application using a TransportClient configured to connect to a
> 2-node ES cluster (i'll leave aside for now why we have to use the
> TransportClient, but it's rationale..)
>
> one of the ES nodes hand a faulty backplane and died.
>
> ES of course kept on trucking with the other node
>
> however since that event the application client has burnt a hell of a lot
> of CPU
>
> which looking at the thread dumps look to be the "New I/O client worker
> #1-5 daemon" style threads used by ES.
>
> I thought somehow with the one ES node dead there's some looping logic
> trying to re-establish connection to it.
>
> so I waited till the Dell guys replaced the backplane and we restored that
> node
>
> once back in green state I was hoping the CPU burn would go away, but alas
> no.
>
> now looking at one of our other instances running in a similar config, I
> note the ES app threads are always runnable because of the NIO, but they're
> generally in a sleep state looking at them.
>
> has anyone else seen this sort of problem?
>
> I'm just gathering a known 'good' thread dump to compare this with.
>
> Here's a gist: [runaway ES client threads · GitHub](https://gist.github.com/1440329)
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [December 7, 2011, 9:03pm UTC](https://discuss.elastic.co/t/runaway-es-transportclient-threads-after-es-node-failure/6083/3 "2011-12-07T21:03:40Z")
Oh geez bad form by me not quoting the version. Yes. 0.17.9 is what we're
using. I'm planning on upgrading to 0.18.x in the next month so that's good
news.
Thanks Shay.
On Thursday, 8 December 2011, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> Heya Paul, which version are you using? It sounds like a problem in netty
> (the networking layer elasticsearch uses) which have been fixed in the
> latest version ([Potential infinite loop in NIO client socket boss thread · Issue #74 · netty/netty · GitHub](https://github.com/netty/netty/issues/74)) and included in
> the latest version of elasticsearch (0.18.5).
>
> On Wed, Dec 7, 2011 at 12:34 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
>
> > I posted this on IRC, but obviously my GMT+11 Timezone is not friendly,
> > so as a backup I post the text here for anyone that might have experience
> > in this:
> > I have an application using a TransportClient configured to connect to a
> > 2-node ES cluster (i'll leave aside for now why we have to use the
> > TransportClient, but it's rationale..)
> > one of the ES nodes hand a faulty backplane and died.
> > ES of course kept on trucking with the other node
> > however since that event the application client has burnt a hell of a
> > lot of CPU
> > which looking at the thread dumps look to be the "New I/O client worker
> > #1-5 daemon" style threads used by ES.
> > I thought somehow with the one ES node dead there's some looping logic
> > trying to re-establish connection to it.
> > so I waited till the Dell guys replaced the backplane and we restored
> > that node
> > once back in green state I was hoping the CPU burn would go away, but
> > alas no.
> > now looking at one of our other instances running in a similar config, I
> > note the ES app threads are always runnable because of the NIO, but they're
> > generally in a sleep state looking at them.
> > has anyone else seen this sort of problem?
> > I'm just gathering a known 'good' thread dump to compare this with.
> > Here's a gist: [https://gist.github.com/1440329](https://gist.github.com/1440329)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:46am UTC](https://discuss.elastic.co/t/runaway-es-transportclient-threads-after-es-node-failure/6083/4 "2017-07-06T03:46:06Z")