# Netty threads eats all of CPU

**URL:** <https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201>\
**Category:** Elasticsearch\
**Created:** [April 7, 2011, 8:10pm UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201 "2011-04-07T20:10:09Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\
**Post date:** [April 7, 2011, 8:10pm UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/1 "2011-04-07T20:10:09Z")

</div>

Hi,  
We are using ES 0.15.2. We use transport client to connect ES server in our  
application. We noticed that our application eats 100% of available CPU.  
When we followed threads in JVM we found two suspicious threads which are

Thread name: CPU USAGE  
New I/O client boss #30 48.736765821  
New I/O client boss #27 48.9538801358

## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?

Mustafa Sener  
[www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\
**Post date:** [April 7, 2011, 8:14pm UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/2 "2011-04-07T20:14:23Z")

</div>

I have following stack trace for thread mentioned

New I/O client boss #30" daemon prio=10 tid=0x0000000046c6f800  
nid=0x46a1 runnable [0x0000000058ad3000..0x0000000058ad3d10]  
java.lang.Thread.State: RUNNABLE

```
at java.util.HashMap$KeySet.iterator(HashMap.java:874)
at java.util.HashSet.iterator(HashSet.java:153)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
- locked <0x00002aaaf3a9c5c0> (a java.util.HashSet)

at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaaf3a9d108> (a sun.nio.ch.Util$1)
- locked <0x00002aaaf3a9d0f0> (a java.util.Collections$UnmodifiableSet)

- locked <0x00002aaaf3a9c530> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)

at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

```

New I/O client boss #27" daemon prio=10 tid=0x0000000046302000  
nid=0x46a3 runnable [0x0000000058bd4000..0x0000000058bd4c10]

java.lang.Thread.State: RUNNABLE  
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)  
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

```
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaaf3a9b5c0> (a sun.nio.ch.Util$1)
- locked <0x00002aaaf3a9b5a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaf3a9a9d0> (a sun.nio.ch.EPollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:619)

```

Locked ownable synchronizers:  
- \<0x00002aaaf3a98970\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

On Thu, Apr 7, 2011 at 11:10 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:

> Hi,  
> We are using ES 0.15.2. We use transport client to connect ES server in our  
> application. We noticed that our application eats 100% of available CPU.  
> When we followed threads in JVM we found two suspicious threads which are
> 
> Thread name: CPU USAGE  
> New I/O client boss #30 48.736765821  
> New I/O client boss #27 48.9538801358
> 
> ## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?
> 
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)

--  
Mustafa Sener  
[www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\
**Post date:** [April 7, 2011, 8:17pm UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/3 "2011-04-07T20:17:22Z")

</div>

Following seems similar problem with mine

[https://issues.apache.org/jira/browse/DIRMINA-678](https://issues.apache.org/jira/browse/DIRMINA-678)

On Thu, Apr 7, 2011 at 11:14 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:

> I have following stack trace for thread mentioned
> 
> New I/O client boss #30" daemon prio=10 tid=0x0000000046c6f800 nid=0x46a1 runnable [0x0000000058ad3000..0x0000000058ad3d10]
> 
> java.lang.Thread.State: RUNNABLE
> 
> at java.util.HashMap$KeySet.iterator(HashMap.java:874)  
> at java.util.HashSet.iterator(HashSet.java:153)  
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> 
> - locked \<0x00002aaaf3a9c5c0\> (a java.util.HashSet)
> 
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)  
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9d108\> (a sun.nio.ch.Util$1)
> 
> - locked \<0x00002aaaf3a9d0f0\> (a java.util.Collections$UnmodifiableSet)
> 
> - locked \<0x00002aaaf3a9c530\> (a sun.nio.ch.EPollSelectorImpl)  
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
> 
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:619)
> 
> New I/O client boss #27" daemon prio=10 tid=0x0000000046302000 nid=0x46a3 runnable [0x0000000058bd4000..0x0000000058bd4c10]
> 
> java.lang.Thread.State: RUNNABLE  
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)  
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> 
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9b5c0\> (a sun.nio.ch.Util$1)
> - locked \<0x00002aaaf3a9b5a8\> (a java.util.Collections$UnmodifiableSet)
> - locked \<0x00002aaaf3a9a9d0\> (a sun.nio.ch.EPollSelectorImpl)
> 
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 
> at java.lang.Thread.run(Thread.java:619)
> 
> Locked ownable synchronizers:
> 
> - \<0x00002aaaf3a98970\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> 
> On Thu, Apr 7, 2011 at 11:10 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:
> 
> > Hi,  
> > We are using ES 0.15.2. We use transport client to connect ES server in  
> > our application. We noticed that our application eats 100% of available CPU.  
> > When we followed threads in JVM we found two suspicious threads which are
> > 
> > Thread name: CPU USAGE  
> > New I/O client boss #30 48.736765821  
> > New I/O client boss #27 48.9538801358
> > 
> > ## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?
> > 
> > Mustafa Sener  
> > [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)

--  
Mustafa Sener  
[www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 7, 2011, 10:02pm UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/4 "2011-04-07T22:02:50Z")

</div>

Heya,

Which version of the jdk are you using? Which vendor? How often does this happen, and does it happen only on the client? You can try and move maybe to the blocking IO mode and see if it solves the problem, we can try and ping Trustin / Netty once we have more info. In order to move to blocking mode, you can set network.tcp.blocking to true.

-shay.banon  
On Thursday, April 7, 2011 at 11:17 PM, Mustafa Sener wrote:

> Following seems similar problem with mine
> 
> [[DIRMINA-678] NioProcessor 100% CPU usage on Linux (epoll selector bug) - ASF JIRA](https://issues.apache.org/jira/browse/DIRMINA-678)
> 
> On Thu, Apr 7, 2011 at 11:14 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com) wrote:
> 
> > I have following stack trace for thread mentioned
> > 
> > New I/O client boss #30" daemon prio=10 tid=0x0000000046c6f800 nid=0x46a1 runnable [0x0000000058ad3000..0x0000000058ad3d10]  
> > java.lang.Thread.State: RUNNABLE  
> > at java.util.HashMap$KeySet.iterator(HashMap.java:874)  
> > at java.util.HashSet.iterator(HashSet.java:153)  
> > at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> > 
> > - locked \<0x00002aaaf3a9c5c0\> (a java.util.HashSet)  
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)  
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > - locked \<0x00002aaaf3a9d108\> (a sun.nio.ch.Util$1)
> > - locked \<0x00002aaaf3a9d0f0\> (a java.util.Collections$UnmodifiableSet)
> > - locked \<0x00002aaaf3a9c530\> (a sun.nio.ch.EPollSelectorImpl)  
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:619)
> > 
> > New I/O client boss #27" daemon prio=10 tid=0x0000000046302000 nid=0x46a3 runnable [0x0000000058bd4000..0x0000000058bd4c10]  
> > java.lang.Thread.State: RUNNABLE  
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)  
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)  
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > 
> > - locked \<0x00002aaaf3a9b5c0\> (a sun.nio.ch.Util$1)
> > - locked \<0x00002aaaf3a9b5a8\> (a java.util.Collections$UnmodifiableSet)
> > - locked \<0x00002aaaf3a9a9d0\> (a sun.nio.ch.EPollSelectorImpl)  
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:619)
> > 
> > Locked ownable synchronizers:
> > 
> > - \<0x00002aaaf3a98970\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > 
> > On Thu, Apr 7, 2011 at 11:10 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com) wrote:
> > 
> > > Hi,  
> > > We are using ES 0.15.2. We use transport client to connect ES server in our application. We noticed that our application eats 100% of available CPU. When we followed threads in JVM we found two suspicious threads which are
> > > 
> > > Thread name: CPU USAGE  
> > > New I/O client boss #30 48.736765821  
> > > New I/O client boss #27 48.9538801358
> > > 
> > > ## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?
> > > 
> > > Mustafa Sener  
> > > [www.ifountain.com](http://www.ifountain.com)
> > 
> > --  
> > Mustafa Sener  
> > [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\
**Post date:** [April 8, 2011, 5:53am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/5 "2011-04-08T05:53:14Z")

</div>

Hi,  
We are using

Java(TM) SE Runtime Environment (build 1.6.0\_11-b03)  
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)

We use ES for a long time this is the first time we come across this problem  
or we may not notify previously.

On Fri, Apr 8, 2011 at 1:02 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Heya,
> 
> Which version of the jdk are you using? Which vendor? How often does this  
> happen, and does it happen only on the client? You can try and move maybe to  
> the blocking IO mode and see if it solves the problem, we can try and ping  
> Trustin / Netty once we have more info. In order to move to blocking mode,  
> you can set network.tcp.blocking to true.
> 
> -shay.banon
> 
> On Thursday, April 7, 2011 at 11:17 PM, Mustafa Sener wrote:
> 
> Following seems similar problem with mine
> 
> [[DIRMINA-678] NioProcessor 100% CPU usage on Linux (epoll selector bug) - ASF JIRA](https://issues.apache.org/jira/browse/DIRMINA-678)
> 
> On Thu, Apr 7, 2011 at 11:14 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:
> 
> I have following stack trace for thread mentioned
> 
> New I/O client boss #30" daemon prio=10 tid=0x0000000046c6f800 nid=0x46a1 runnable [0x0000000058ad3000..0x0000000058ad3d10]
> 
> java.lang.Thread.State: RUNNABLE
> 
> at java.util.HashMap$KeySet.iterator(HashMap.java:874)  
> at java.util.HashSet.iterator(HashSet.java:153)  
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> 
> - locked \<0x00002aaaf3a9c5c0\> (a java.util.HashSet)
> 
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)  
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9d108\> (a sun.nio.ch.Util$1)
> 
> - locked \<0x00002aaaf3a9d0f0\> (a java.util.Collections$UnmodifiableSet)
> 
> - locked \<0x00002aaaf3a9c530\> (a sun.nio.ch.EPollSelectorImpl)  
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
> 
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:619)
> 
> New I/O client boss #27" daemon prio=10 tid=0x0000000046302000 nid=0x46a3 runnable [0x0000000058bd4000..0x0000000058bd4c10]
> 
> java.lang.Thread.State: RUNNABLE  
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)  
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> 
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9b5c0\> (a sun.nio.ch.Util$1)
> - locked \<0x00002aaaf3a9b5a8\> (a java.util.Collections$UnmodifiableSet)
> - locked \<0x00002aaaf3a9a9d0\> (a sun.nio.ch.EPollSelectorImpl)
> 
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 
> at java.lang.Thread.run(Thread.java:619)
> 
> Locked ownable synchronizers:
> 
> - \<0x00002aaaf3a98970\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> 
> On Thu, Apr 7, 2011 at 11:10 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:
> 
> Hi,  
> We are using ES 0.15.2. We use transport client to connect ES server in our  
> application. We noticed that our application eats 100% of available CPU.  
> When we followed threads in JVM we found two suspicious threads which are
> 
> Thread name: CPU USAGE  
> New I/O client boss #30 48.736765821  
> New I/O client boss #27 48.9538801358
> 
> ## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?
> 
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)

--  
Mustafa Sener  
[www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 8, 2011, 8:52am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/6 "2011-04-08T08:52:57Z")

</div>

Heya,

So, you might have hit the bug that was fixed in the jdk 1.6u18, any reason why you don't use a newer version?

-shay.banon  
On Friday, April 8, 2011 at 8:53 AM, Mustafa Sener wrote:

> Hi,  
> We are using
> 
> Java(TM) SE Runtime Environment (build 1.6.0\_11-b03)  
> Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)
> 
> We use ES for a long time this is the first time we come across this problem or we may not notify previously.
> 
> On Fri, Apr 8, 2011 at 1:02 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > Heya,
> > 
> > Which version of the jdk are you using? Which vendor? How often does this happen, and does it happen only on the client? You can try and move maybe to the blocking IO mode and see if it solves the problem, we can try and ping Trustin / Netty once we have more info. In order to move to blocking mode, you can set network.tcp.blocking to true.
> > 
> > -shay.banon  
> > On Thursday, April 7, 2011 at 11:17 PM, Mustafa Sener wrote:
> > 
> > > Following seems similar problem with mine
> > > 
> > > [[DIRMINA-678] NioProcessor 100% CPU usage on Linux (epoll selector bug) - ASF JIRA](https://issues.apache.org/jira/browse/DIRMINA-678)
> > > 
> > > On Thu, Apr 7, 2011 at 11:14 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com) wrote:
> > > 
> > > > I have following stack trace for thread mentioned
> > > > 
> > > > New I/O client boss #30" daemon prio=10 tid=0x0000000046c6f800 nid=0x46a1 runnable [0x0000000058ad3000..0x0000000058ad3d10]  
> > > > java.lang.Thread.State: RUNNABLE  
> > > > at java.util.HashMap$KeySet.iterator(HashMap.java:874)  
> > > > at java.util.HashSet.iterator(HashSet.java:153)  
> > > > at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> > > > 
> > > > - locked \<0x00002aaaf3a9c5c0\> (a java.util.HashSet)  
> > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)  
> > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > - locked \<0x00002aaaf3a9d108\> (a sun.nio.ch.Util$1)
> > > > - locked \<0x00002aaaf3a9d0f0\> (a java.util.Collections$UnmodifiableSet)
> > > > - locked \<0x00002aaaf3a9c530\> (a sun.nio.ch.EPollSelectorImpl)  
> > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> > > > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> > > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > > > at java.lang.Thread.run(Thread.java:619)
> > > > 
> > > > New I/O client boss #27" daemon prio=10 tid=0x0000000046302000 nid=0x46a3 runnable [0x0000000058bd4000..0x0000000058bd4c10]  
> > > > java.lang.Thread.State: RUNNABLE  
> > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
> > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)  
> > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)  
> > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > 
> > > > - locked \<0x00002aaaf3a9b5c0\> (a sun.nio.ch.Util$1)
> > > > - locked \<0x00002aaaf3a9b5a8\> (a java.util.Collections$UnmodifiableSet)
> > > > - locked \<0x00002aaaf3a9a9d0\> (a sun.nio.ch.EPollSelectorImpl)  
> > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> > > > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> > > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > > > at java.lang.Thread.run(Thread.java:619)
> > > > 
> > > > Locked ownable synchronizers:
> > > > 
> > > > - \<0x00002aaaf3a98970\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > > > 
> > > > On Thu, Apr 7, 2011 at 11:10 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com) wrote:
> > > > 
> > > > > Hi,  
> > > > > We are using ES 0.15.2. We use transport client to connect ES server in our application. We noticed that our application eats 100% of available CPU. When we followed threads in JVM we found two suspicious threads which are
> > > > > 
> > > > > Thread name: CPU USAGE  
> > > > > New I/O client boss #30 48.736765821  
> > > > > New I/O client boss #27 48.9538801358
> > > > > 
> > > > > ## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?
> > > > > 
> > > > > Mustafa Sener  
> > > > > [www.ifountain.com](http://www.ifountain.com)
> > > > 
> > > > --  
> > > > Mustafa Sener  
> > > > [www.ifountain.com](http://www.ifountain.com)
> > > 
> > > --  
> > > Mustafa Sener  
> > > [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\
**Post date:** [April 8, 2011, 9:59am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/7 "2011-04-08T09:59:51Z")

</div>

No we can upgrade it. But I just want to warn you about this problem.

On Fri, Apr 8, 2011 at 11:52 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Heya,
> 
> So, you might have hit the bug that was fixed in the jdk 1.6u18, any  
> reason why you don't use a newer version?
> 
> -shay.banon
> 
> On Friday, April 8, 2011 at 8:53 AM, Mustafa Sener wrote:
> 
> Hi,  
> We are using
> 
> Java(TM) SE Runtime Environment (build 1.6.0\_11-b03)  
> Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)
> 
> We use ES for a long time this is the first time we come across this  
> problem or we may not notify previously.
> 
> On Fri, Apr 8, 2011 at 1:02 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> Heya,
> 
> Which version of the jdk are you using? Which vendor? How often does this  
> happen, and does it happen only on the client? You can try and move maybe to  
> the blocking IO mode and see if it solves the problem, we can try and ping  
> Trustin / Netty once we have more info. In order to move to blocking mode,  
> you can set network.tcp.blocking to true.
> 
> -shay.banon
> 
> On Thursday, April 7, 2011 at 11:17 PM, Mustafa Sener wrote:
> 
> Following seems similar problem with mine
> 
> [[DIRMINA-678] NioProcessor 100% CPU usage on Linux (epoll selector bug) - ASF JIRA](https://issues.apache.org/jira/browse/DIRMINA-678)
> 
> On Thu, Apr 7, 2011 at 11:14 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:
> 
> I have following stack trace for thread mentioned
> 
> New I/O client boss #30" daemon prio=10 tid=0x0000000046c6f800 nid=0x46a1 runnable [0x0000000058ad3000..0x0000000058ad3d10]
> 
> java.lang.Thread.State: RUNNABLE
> 
> at java.util.HashMap$KeySet.iterator(HashMap.java:874)  
> at java.util.HashSet.iterator(HashSet.java:153)  
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> 
> - locked \<0x00002aaaf3a9c5c0\> (a java.util.HashSet)
> 
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)  
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9d108\> (a sun.nio.ch.Util$1)
> 
> - locked \<0x00002aaaf3a9d0f0\> (a java.util.Collections$UnmodifiableSet)
> 
> - locked \<0x00002aaaf3a9c530\> (a sun.nio.ch.EPollSelectorImpl)  
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
> 
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:619)
> 
> New I/O client boss #27" daemon prio=10 tid=0x0000000046302000 nid=0x46a3 runnable [0x0000000058bd4000..0x0000000058bd4c10]
> 
> java.lang.Thread.State: RUNNABLE  
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)  
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> 
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9b5c0\> (a sun.nio.ch.Util$1)
> - locked \<0x00002aaaf3a9b5a8\> (a java.util.Collections$UnmodifiableSet)
> - locked \<0x00002aaaf3a9a9d0\> (a sun.nio.ch.EPollSelectorImpl)
> 
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 
> at java.lang.Thread.run(Thread.java:619)
> 
> Locked ownable synchronizers:
> 
> - \<0x00002aaaf3a98970\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> 
> On Thu, Apr 7, 2011 at 11:10 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:
> 
> Hi,  
> We are using ES 0.15.2. We use transport client to connect ES server in our  
> application. We noticed that our application eats 100% of available CPU.  
> When we followed threads in JVM we found two suspicious threads which are
> 
> Thread name: CPU USAGE  
> New I/O client boss #30 48.736765821  
> New I/O client boss #27 48.9538801358
> 
> ## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?
> 
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)

--  
Mustafa Sener  
[www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\
**Post date:** [April 8, 2011, 10:24am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/8 "2011-04-08T10:24:30Z")

</div>

Shay,

Is that "network.tcp.blocking" something new in ES? I cannot find it in the  
docs.

Regards.

On Fri, Apr 8, 2011 at 12:02 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Heya,
> 
> Which version of the jdk are you using? Which vendor? How often does this  
> happen, and does it happen only on the client? You can try and move maybe to  
> the blocking IO mode and see if it solves the problem, we can try and ping  
> Trustin / Netty once we have more info. In order to move to blocking mode,  
> you can set network.tcp.blocking to true.
> 
> -shay.banon
> 
> On Thursday, April 7, 2011 at 11:17 PM, Mustafa Sener wrote:
> 
> Following seems similar problem with mine
> 
> [[DIRMINA-678] NioProcessor 100% CPU usage on Linux (epoll selector bug) - ASF JIRA](https://issues.apache.org/jira/browse/DIRMINA-678)
> 
> On Thu, Apr 7, 2011 at 11:14 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:
> 
> I have following stack trace for thread mentioned
> 
> New I/O client boss #30" daemon prio=10 tid=0x0000000046c6f800 nid=0x46a1 runnable [0x0000000058ad3000..0x0000000058ad3d10]
> 
> java.lang.Thread.State: RUNNABLE
> 
> at java.util.HashMap$KeySet.iterator(HashMap.java:874)  
> at java.util.HashSet.iterator(HashSet.java:153)  
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> 
> - locked \<0x00002aaaf3a9c5c0\> (a java.util.HashSet)
> 
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)  
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9d108\> (a sun.nio.ch.Util$1)
> 
> - locked \<0x00002aaaf3a9d0f0\> (a java.util.Collections$UnmodifiableSet)
> 
> - locked \<0x00002aaaf3a9c530\> (a sun.nio.ch.EPollSelectorImpl)  
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
> 
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:619)
> 
> New I/O client boss #27" daemon prio=10 tid=0x0000000046302000 nid=0x46a3 runnable [0x0000000058bd4000..0x0000000058bd4c10]
> 
> java.lang.Thread.State: RUNNABLE  
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)  
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> 
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9b5c0\> (a sun.nio.ch.Util$1)
> - locked \<0x00002aaaf3a9b5a8\> (a java.util.Collections$UnmodifiableSet)
> - locked \<0x00002aaaf3a9a9d0\> (a sun.nio.ch.EPollSelectorImpl)
> 
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 
> at java.lang.Thread.run(Thread.java:619)
> 
> Locked ownable synchronizers:
> 
> - \<0x00002aaaf3a98970\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> 
> On Thu, Apr 7, 2011 at 11:10 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:
> 
> Hi,  
> We are using ES 0.15.2. We use transport client to connect ES server in our  
> application. We noticed that our application eats 100% of available CPU.  
> When we followed threads in JVM we found two suspicious threads which are
> 
> Thread name: CPU USAGE  
> New I/O client boss #30 48.736765821  
> New I/O client boss #27 48.9538801358
> 
> ## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?
> 
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 8, 2011, 11:31am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/9 "2011-04-08T11:31:26Z")

</div>

The `network.tcp.blocking` is not in the docs, I added it to test perf between blocking and non blocking, and kept it there. It is recommended not to use it (well, unless there are bugs with nio).  
On Friday, April 8, 2011 at 1:24 PM, Enrique Medina Montenegro wrote:

> Shay,
> 
> Is that "network.tcp.blocking" something new in ES? I cannot find it in the docs.
> 
> Regards.
> 
> On Fri, Apr 8, 2011 at 12:02 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > Heya,
> > 
> > Which version of the jdk are you using? Which vendor? How often does this happen, and does it happen only on the client? You can try and move maybe to the blocking IO mode and see if it solves the problem, we can try and ping Trustin / Netty once we have more info. In order to move to blocking mode, you can set network.tcp.blocking to true.
> > 
> > -shay.banon  
> > On Thursday, April 7, 2011 at 11:17 PM, Mustafa Sener wrote:
> > 
> > > Following seems similar problem with mine
> > > 
> > > [[DIRMINA-678] NioProcessor 100% CPU usage on Linux (epoll selector bug) - ASF JIRA](https://issues.apache.org/jira/browse/DIRMINA-678)
> > > 
> > > On Thu, Apr 7, 2011 at 11:14 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com) wrote:
> > > 
> > > > I have following stack trace for thread mentioned
> > > > 
> > > > New I/O client boss #30" daemon prio=10 tid=0x0000000046c6f800 nid=0x46a1 runnable [0x0000000058ad3000..0x0000000058ad3d10]  
> > > > java.lang.Thread.State: RUNNABLE  
> > > > at java.util.HashMap$KeySet.iterator(HashMap.java:874)  
> > > > at java.util.HashSet.iterator(HashSet.java:153)  
> > > > at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> > > > 
> > > > - locked \<0x00002aaaf3a9c5c0\> (a java.util.HashSet)  
> > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)  
> > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > - locked \<0x00002aaaf3a9d108\> (a sun.nio.ch.Util$1)
> > > > - locked \<0x00002aaaf3a9d0f0\> (a java.util.Collections$UnmodifiableSet)
> > > > - locked \<0x00002aaaf3a9c530\> (a sun.nio.ch.EPollSelectorImpl)  
> > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> > > > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> > > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > > > at java.lang.Thread.run(Thread.java:619)
> > > > 
> > > > New I/O client boss #27" daemon prio=10 tid=0x0000000046302000 nid=0x46a3 runnable [0x0000000058bd4000..0x0000000058bd4c10]  
> > > > java.lang.Thread.State: RUNNABLE  
> > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
> > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)  
> > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)  
> > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > 
> > > > - locked \<0x00002aaaf3a9b5c0\> (a sun.nio.ch.Util$1)
> > > > - locked \<0x00002aaaf3a9b5a8\> (a java.util.Collections$UnmodifiableSet)
> > > > - locked \<0x00002aaaf3a9a9d0\> (a sun.nio.ch.EPollSelectorImpl)  
> > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> > > > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> > > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > > > at java.lang.Thread.run(Thread.java:619)
> > > > 
> > > > Locked ownable synchronizers:
> > > > 
> > > > - \<0x00002aaaf3a98970\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > > > 
> > > > On Thu, Apr 7, 2011 at 11:10 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com) wrote:
> > > > 
> > > > > Hi,  
> > > > > We are using ES 0.15.2. We use transport client to connect ES server in our application. We noticed that our application eats 100% of available CPU. When we followed threads in JVM we found two suspicious threads which are
> > > > > 
> > > > > Thread name: CPU USAGE  
> > > > > New I/O client boss #30 48.736765821  
> > > > > New I/O client boss #27 48.9538801358
> > > > > 
> > > > > ## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?
> > > > > 
> > > > > Mustafa Sener  
> > > > > [www.ifountain.com](http://www.ifountain.com)
> > > > 
> > > > --  
> > > > Mustafa Sener  
> > > > [www.ifountain.com](http://www.ifountain.com)
> > > 
> > > --  
> > > Mustafa Sener  
> > > [www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\
**Post date:** [April 8, 2011, 11:38am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/10 "2011-04-08T11:38:43Z")

</div>

Understood 🙂

On Fri, Apr 8, 2011 at 1:31 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> The `network.tcp.blocking` is not in the docs, I added it to test perf  
> between blocking and non blocking, and kept it there. It is recommended not  
> to use it (well, unless there are bugs with nio).
> 
> On Friday, April 8, 2011 at 1:24 PM, Enrique Medina Montenegro wrote:
> 
> Shay,
> 
> Is that "network.tcp.blocking" something new in ES? I cannot find it in  
> the docs.
> 
> Regards.
> 
> On Fri, Apr 8, 2011 at 12:02 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> Heya,
> 
> Which version of the jdk are you using? Which vendor? How often does this  
> happen, and does it happen only on the client? You can try and move maybe to  
> the blocking IO mode and see if it solves the problem, we can try and ping  
> Trustin / Netty once we have more info. In order to move to blocking mode,  
> you can set network.tcp.blocking to true.
> 
> -shay.banon
> 
> On Thursday, April 7, 2011 at 11:17 PM, Mustafa Sener wrote:
> 
> Following seems similar problem with mine
> 
> [[DIRMINA-678] NioProcessor 100% CPU usage on Linux (epoll selector bug) - ASF JIRA](https://issues.apache.org/jira/browse/DIRMINA-678)
> 
> On Thu, Apr 7, 2011 at 11:14 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:
> 
> I have following stack trace for thread mentioned
> 
> New I/O client boss #30" daemon prio=10 tid=0x0000000046c6f800 nid=0x46a1 runnable [0x0000000058ad3000..0x0000000058ad3d10]
> 
> java.lang.Thread.State: RUNNABLE
> 
> at java.util.HashMap$KeySet.iterator(HashMap.java:874)  
> at java.util.HashSet.iterator(HashSet.java:153)  
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> 
> - locked \<0x00002aaaf3a9c5c0\> (a java.util.HashSet)
> 
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)  
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9d108\> (a sun.nio.ch.Util$1)
> 
> - locked \<0x00002aaaf3a9d0f0\> (a java.util.Collections$UnmodifiableSet)
> 
> - locked \<0x00002aaaf3a9c530\> (a sun.nio.ch.EPollSelectorImpl)  
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
> 
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:619)
> 
> New I/O client boss #27" daemon prio=10 tid=0x0000000046302000 nid=0x46a3 runnable [0x0000000058bd4000..0x0000000058bd4c10]
> 
> java.lang.Thread.State: RUNNABLE  
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)  
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> 
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 
> - locked \<0x00002aaaf3a9b5c0\> (a sun.nio.ch.Util$1)
> - locked \<0x00002aaaf3a9b5a8\> (a java.util.Collections$UnmodifiableSet)
> - locked \<0x00002aaaf3a9a9d0\> (a sun.nio.ch.EPollSelectorImpl)
> 
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)  
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 
> at java.lang.Thread.run(Thread.java:619)
> 
> Locked ownable synchronizers:
> 
> - \<0x00002aaaf3a98970\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> 
> On Thu, Apr 7, 2011 at 11:10 PM, Mustafa Sener [mustafa.sener@gmail.com](mailto:mustafa.sener@gmail.com)wrote:
> 
> Hi,  
> We are using ES 0.15.2. We use transport client to connect ES server in our  
> application. We noticed that our application eats 100% of available CPU.  
> When we followed threads in JVM we found two suspicious threads which are
> 
> Thread name: CPU USAGE  
> New I/O client boss #30 48.736765821  
> New I/O client boss #27 48.9538801358
> 
> ## Do you have any ideas about this problem? I searched about this problem and found some other people using Netty come across with same problem. Do you have any suggestion to fix this issue?
> 
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)
> 
> --  
> Mustafa Sener  
> [www.ifountain.com](http://www.ifountain.com)

---

<div class="post-metadata">

**Author:** ![jigarjm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jigarjm/32/858_2.png) [@jigarjm](https://discuss.elastic.co/u/jigarjm)\
**Post date:** [March 5, 2015, 2:06am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/11 "2015-03-05T02:06:55Z")

</div>

I ran into this bug again

ES is running on : JDK 1.8.0\_31\<nabble\_embed\>\</nabble\_embed\>  
Client is running on : JDK 1.7.0\_45  
Elasticsearch version: 1.4.3

lots of threads are waiting at in client and on server both

\<nabble\_embed\>

java.lang.Thread.State: RUNNABLE  
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)  
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)  
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)  
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)  
- locked \<0x00007f45623d2090\> (a sun.nio.ch.Util$2)  
- locked \<0x00007f45623d2078\> (a java.util.Collections$UnmodifiableSet)  
- locked \<0x00007f45623d1eb8\> (a sun.nio.ch.EPollSelectorImpl)  
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)  
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:341)  
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:189)  
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute(CloseableHttpAsyncClientBase.java:67)  
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000(CloseableHttpAsyncClientBase.java:38)  
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:57)  
at java.lang.Thread.run(Thread.java:744)

\</nabble\_embed\>

App is doing heavy writes, and it stays stuck in here

$ ~/tools/jdk1.8.0\_31/bin/jstack 6142 | grep 'EPollArrayWrapper.epollWait' | wc -l  
195

any suggestions ?

this is thread pool related config

threadpool.bulk.type: fixed  
threadpool.bulk.size: 100  
threadpool.bulk.queue\_size: 300

threadpool.index.type: fixed  
threadpool.index.size: 100  
threadpool.index.queue\_size: 100

indices.memory.index\_buffer\_size: 30%  
indices.memory.min\_shard\_index\_buffer\_size: 1024mb  
indices.memory.min\_index\_buffer\_size: 1024mb

Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:28am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4201/12 "2017-07-06T00:28:25Z")

</div>