Netty threads eats all of CPU

Elastic Stack Elasticsearch
1

Hi,
Several weeks ago I reported same issue.
http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by
Shay. But the problem occurred again. The problem is on TransportClient side
not on ES server. It seems that it is related to following threads which
eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800
nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
- locked <0x00000007871eadf0> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800
nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
- locked <0x000000078721e8d8> (a java.lang.Object)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800
nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
- locked <0x00000007871eadf0> (a java.lang.Object)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800
nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
- locked <0x000000078721e8d8> (a java.lang.Object)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800
nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap$KeySet.iterator(HashMap.java:874)
at java.util.HashSet.iterator(HashSet.java:153)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
- locked <0x00000007871e5538> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800
nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
- locked <0x000000078721e8d8> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800
nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
- locked <0x00000007871eadf0> (a java.lang.Object)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800
nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

--
Mustafa Sener
www.ifountain.com

2

Can you try and move to OIO in the transport client and see if it still happens?
On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote:

Hi,
Several weeks ago I reported same issue.
http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked <0x00000007871eadf0> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$B
oss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x000000078721e8d8> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x00
0000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) =====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x00000007871eadf0> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Th
readPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x000000078721e8d8> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at java.util.HashMap$KeySet.iterator(HashMap.java:874) at java.util.HashSet.iterator(HashSet.java:153) at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127) - locked <0x00000007871e5538> (a java.util.HashSet) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.
netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked <0x000000078721e8d8> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:

  1. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)====================================
    ================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x00000007871eadf0> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Th
readPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a jav
a.util.concurrent.locks.ReentrantLock$NonfairSync)

Mustafa Sener
www.ifountain.com

3

what do you mean by O|O?

On Fri, May 6, 2011 at 6:54 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Can you try and move to OIO in the transport client and see if it still
happens?

On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote:

Hi,
Several weeks ago I reported same issue.

http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by
Shay. But the problem occurred again. The problem is on TransportClient side
not on ES server. It seems that it is related to following threads which
eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)

  • locked <0x00000007871eadf0> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
    at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
  • locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
  • locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)

  • locked <0x000000078721e8d8> (a java.lang.Object)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x000000078721e730> (a sun.nio.ch.Util$2)
  • locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
  • locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    =====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)

  • locked <0x00000007871eadf0> (a java.lang.Object)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
  • locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
  • locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)

  • locked <0x000000078721e8d8> (a java.lang.Object)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x000000078721e730> (a sun.nio.ch.Util$2)
  • locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
  • locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap$KeySet.iterator(HashMap.java:874)
at java.util.HashSet.iterator(HashSet.java:153)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)

  • locked <0x00000007871e5538> (a java.util.HashSet)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
  • locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
  • locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)

  • locked <0x000000078721e8d8> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
    at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x000000078721e730> (a sun.nio.ch.Util$2)
  • locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
  • locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)

  • locked <0x00000007871eadf0> (a java.lang.Object)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
  • locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
  • locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com
WebRep
Overall rating

4

Sorry, thats the netty lingo. Set network.tcp.blocking setting to true.
On Friday, May 6, 2011 at 7:01 PM, Mustafa Sener wrote:

what do you mean by O|O?

On Fri, May 6, 2011 at 6:54 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Can you try and move to OIO in the transport client and see if it still happens?
On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote:

Hi,
Several weeks ago I reported same issue.
http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked <0x00000007871eadf0> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSi
nk$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x000000078721e8d8> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <
0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) =====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x00000007871eadf0> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTas
k(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x000000078721e8d8> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at java.util.HashMap$KeySet.iterator(HashMap.java:874) at java.util.HashSet.iterator(HashSet.java:153) at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127) - locked <0x00000007871e5538> (a java.util.HashSet) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.com
mon.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175) - locked <0x000000078721e8d8> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.j
ava:64) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)================================
====================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x00000007871eadf0> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTas
k(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)

Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com
WebRep

Overall rating

5

I applied setting to TransportClient. I will notify you if this problem
occurs again.

Thanks.

On Fri, May 6, 2011 at 7:06 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Sorry, thats the netty lingo. Set network.tcp.blocking setting to true.

On Friday, May 6, 2011 at 7:01 PM, Mustafa Sener wrote:

what do you mean by O|O?

On Fri, May 6, 2011 at 6:54 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Can you try and move to OIO in the transport client and see if it still
happens?

On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote:

Hi,
Several weeks ago I reported same issue.

http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by
Shay. But the problem occurred again. The problem is on TransportClient side
not on ES server. It seems that it is related to following threads which
eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)

  • locked <0x00000007871eadf0> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
    at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
  • locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
  • locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)

  • locked <0x000000078721e8d8> (a java.lang.Object)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x000000078721e730> (a sun.nio.ch.Util$2)
  • locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
  • locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    =====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)

  • locked <0x00000007871eadf0> (a java.lang.Object)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
  • locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
  • locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)

  • locked <0x000000078721e8d8> (a java.lang.Object)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x000000078721e730> (a sun.nio.ch.Util$2)
  • locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
  • locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap$KeySet.iterator(HashMap.java:874)
at java.util.HashSet.iterator(HashSet.java:153)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)

  • locked <0x00000007871e5538> (a java.util.HashSet)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
  • locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
  • locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)

  • locked <0x000000078721e8d8> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
    at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x000000078721e730> (a sun.nio.ch.Util$2)
  • locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
  • locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)

  • locked <0x00000007871eadf0> (a java.lang.Object)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  • locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
  • locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
  • locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
java.lang.Thread.State: RUNNABLE
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

  • <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com
WebRep
Overall rating

--
Mustafa Sener
www.ifountain.com