# Netty threads eats all of CPU **URL:** \ **Category:** Elasticsearch\ **Created:** [May 6, 2011, 9:39am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4357 "2011-05-06T09:39:45Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\ **Post date:** [May 6, 2011, 9:39am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4357/1 "2011-05-06T09:39:45Z")
Hi, Several weeks ago I reported same issue. [http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html](http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html) We upgraded JRE to latest one which is 1.6.0\_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU: New I/O client boss #5 New I/O client boss #7 # I took multiple stack traces at different times "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked \<0x00000007871eadf0\> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x000000078721e8d8\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) # Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x00000007871eadf0\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x000000078721e8d8\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) ======================================================================================= "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at java.util.HashMap$KeySet.iterator(HashMap.java:874) at java.util.HashSet.iterator(HashSet.java:153) at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127) - locked \<0x00000007871e5538\> (a java.util.HashSet) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked \<0x000000078721e8d8\> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) ==================================================================== "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x00000007871eadf0\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) -- Mustafa Sener [www.ifountain.com](http://www.ifountain.com) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 6, 2011, 3:54pm UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4357/2 "2011-05-06T15:54:48Z")
Can you try and move to OIO in the transport client and see if it still happens? On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote: > Hi, > Several weeks ago I reported same issue. > [http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html](http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html) > > We upgraded JRE to latest one which is 1.6.0\_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU: > > New I/O client boss #5 > New I/O client boss #7 > > # I took multiple stack traces at different times > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked \<0x00000007871eadf0\> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$B > oss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x000000078721e8d8\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) - locked \<0x00 > 0000078721e720\> (a java.util.Collections$UnmodifiableSet) - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) ===================================================================== > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x00000007871eadf0\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Th > readPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x000000078721e8d8\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > ======================================================================================= > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at java.util.HashMap$KeySet.iterator(HashMap.java:874) at java.util.HashSet.iterator(HashSet.java:153) at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127) - locked \<0x00000007871e5538\> (a java.util.HashSet) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common. > netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked \<0x000000078721e8d8\> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java: 1. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)==================================== ================================ > ## "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x00000007871eadf0\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Th readPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a jav a.util.concurrent.locks.ReentrantLock$NonfairSync) > > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) ---
**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\ **Post date:** [May 6, 2011, 4:01pm UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4357/3 "2011-05-06T16:01:03Z")
what do you mean by O|O? On Fri, May 6, 2011 at 6:54 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Can you try and move to OIO in the transport client and see if it still > happens? > > On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote: > > Hi, > Several weeks ago I reported same issue. > > [http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html](http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html) > > We upgraded JRE to latest one which is 1.6.0\_24-b07 as it is suggested by > Shay. But the problem occurred again. The problem is on TransportClient side > not on ES server. It seems that it is related to following threads which > eats all CPU: > > New I/O client boss #5 > New I/O client boss #7 > > # I took multiple stack traces at different times > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) > at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) > at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) > > - locked \<0x00000007871eadf0\> (a java.lang.Object) > at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) > at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) > - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.IOUtil.drain(Native Method) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) > > - locked \<0x000000078721e8d8\> (a java.lang.Object) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) > - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > ===================================================================== > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.IOUtil.drain(Native Method) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) > > - locked \<0x00000007871eadf0\> (a java.lang.Object) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) > - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.IOUtil.drain(Native Method) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) > > - locked \<0x000000078721e8d8\> (a java.lang.Object) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) > - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > ======================================================================================= > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] > java.lang.Thread.State: RUNNABLE > at java.util.HashMap$KeySet.iterator(HashMap.java:874) > at java.util.HashSet.iterator(HashSet.java:153) > at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127) > > - locked \<0x00000007871e5538\> (a java.util.HashSet) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) > - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) > at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) > at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) > > - locked \<0x000000078721e8d8\> (a java.lang.Object) > at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) > at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) > - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > ==================================================================== > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.IOUtil.drain(Native Method) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) > > - locked \<0x00000007871eadf0\> (a java.lang.Object) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) > - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] > java.lang.Thread.State: RUNNABLE > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) -- Mustafa Sener [www.ifountain.com](http://www.ifountain.com) WebRep Overall rating ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 6, 2011, 4:06pm UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4357/4 "2011-05-06T16:06:11Z")
Sorry, thats the netty lingo. Set network.tcp.blocking setting to true. On Friday, May 6, 2011 at 7:01 PM, Mustafa Sener wrote: > what do you mean by O|O? > > On Fri, May 6, 2011 at 6:54 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Can you try and move to OIO in the transport client and see if it still happens? > > On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote: > > > > > Hi, > > > Several weeks ago I reported same issue. > > > [http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html](http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html) > > > > > > We upgraded JRE to latest one which is 1.6.0\_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU: > > > > > > New I/O client boss #5 > > > New I/O client boss #7 > > > > > > # I took multiple stack traces at different times > > > > > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked \<0x00000007871eadf0\> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSi > > > nk$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x000000078721e8d8\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) - locked \< > > > 0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) ===================================================================== > > > > > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x00000007871eadf0\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTas > > > k(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x000000078721e8d8\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > > > > > ======================================================================================= > > > > > > ## "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at java.util.HashMap$KeySet.iterator(HashMap.java:874) at java.util.HashSet.iterator(HashSet.java:153) at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127) - locked \<0x00000007871e5538\> (a java.util.HashSet) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at [org.elasticsearch.com](http://org.elasticsearch.com) mon.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked \<0x000000078721e8d8\> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.j ava:64) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)================================ ==================================== "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked \<0x00000007871eadf0\> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTas k(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > > > > > Mustafa Sener > > > [www.ifountain.com](http://www.ifountain.com) > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) > WebRep > > Overall rating ---
**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\ **Post date:** [May 10, 2011, 7:20am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4357/5 "2011-05-10T07:20:51Z")
I applied setting to TransportClient. I will notify you if this problem occurs again. Thanks. On Fri, May 6, 2011 at 7:06 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Sorry, thats the netty lingo. Set network.tcp.blocking setting to true. > > On Friday, May 6, 2011 at 7:01 PM, Mustafa Sener wrote: > > what do you mean by O|O? > > On Fri, May 6, 2011 at 6:54 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Can you try and move to OIO in the transport client and see if it still > happens? > > On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote: > > Hi, > Several weeks ago I reported same issue. > > [http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html](http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html) > > We upgraded JRE to latest one which is 1.6.0\_24-b07 as it is suggested by > Shay. But the problem occurred again. The problem is on TransportClient side > not on ES server. It seems that it is related to following threads which > eats all CPU: > > New I/O client boss #5 > New I/O client boss #7 > > # I took multiple stack traces at different times > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) > at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) > at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) > > - locked \<0x00000007871eadf0\> (a java.lang.Object) > at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) > at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) > - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.IOUtil.drain(Native Method) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) > > - locked \<0x000000078721e8d8\> (a java.lang.Object) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) > - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > ===================================================================== > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.IOUtil.drain(Native Method) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) > > - locked \<0x00000007871eadf0\> (a java.lang.Object) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) > - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.IOUtil.drain(Native Method) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) > > - locked \<0x000000078721e8d8\> (a java.lang.Object) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) > - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > ======================================================================================= > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] > java.lang.Thread.State: RUNNABLE > at java.util.HashMap$KeySet.iterator(HashMap.java:874) > at java.util.HashSet.iterator(HashSet.java:153) > at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127) > > - locked \<0x00000007871e5538\> (a java.util.HashSet) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) > - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) > at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) > at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) > > - locked \<0x000000078721e8d8\> (a java.lang.Object) > at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) > at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000078721e730\> (a sun.nio.ch.Util$2) > - locked \<0x000000078721e720\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x000000078721df90\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > ==================================================================== > > "New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.IOUtil.drain(Native Method) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) > > - locked \<0x00000007871eadf0\> (a java.lang.Object) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x00000007871eac48\> (a sun.nio.ch.Util$2) > - locked \<0x00000007871eac38\> (a java.util.Collections$UnmodifiableSet) > - locked \<0x00000007871e54e0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x00000007871eae58\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] > java.lang.Thread.State: RUNNABLE > at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > > - \<0x000000078721dc08\> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) > WebRep > Overall rating -- Mustafa Sener [www.ifountain.com](http://www.ifountain.com) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:06am UTC](https://discuss.elastic.co/t/netty-threads-eats-all-of-cpu/4357/6 "2017-07-06T04:06:35Z")