Every once in a while a single CPU will run 100% for an extended period of time

Elastic Stack Elasticsearch
1

I am running Elasticsearch 0.15.2 on EC2 large instances. Every once
in a while a single CPU will run 100% for an extended period of time.
I cannot figure out what it is doing. At first I thought it might be
running a merge, but after dumping the stacktraces while it was
happening, the only merge related information I can find is this:

"elasticsearch[merge]-pool-6-thread-6602" daemon prio=10
tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition
[0x0000000044068000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x000000060c170fb8> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer
    $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:

at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
947)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

So it doesnt appear to be doing a merge. Any thoughts?

Thanks

2

Do you see any other thread busy?

p.s. Upgrade.

On Mon, Nov 21, 2011 at 5:40 PM, eroyal egan@spredfast.com wrote:

I am running Elasticsearch 0.15.2 on EC2 large instances. Every once
in a while a single CPU will run 100% for an extended period of time.
I cannot figure out what it is doing. At first I thought it might be
running a merge, but after dumping the stacktraces while it was
happening, the only merge related information I can find is this:

"elasticsearch[merge]-pool-6-thread-6602" daemon prio=10
tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition
[0x0000000044068000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x000000060c170fb8> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer
    $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:

at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
947)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

So it doesnt appear to be doing a merge. Any thoughts?

Thanks

3

Yeah there are several threads that are busy... I'd just like to
figure out which one(s) is running that much CPU for that long. I
dont 'think' it's a query as our performance logs dont show any query
taking that long. It also doesn't behave like a GC problem. These
are the runnable threads:

"Attach Listener" daemon prio=10 tid=0x00002aaae935e800 nid=0x6125
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"elasticsearch[cached]-pool-1-thread-31448" daemon prio=10
tid=0x00002aab1fbfd000 nid=0x5f19 runnable [0x000000004163e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610)
at org.apache.lucene.store.NIOFSDirectory
$NIOFSIndexInput.readInternal(NIOFSDirectory.java:168)
at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
160)
at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:
39)
at org.apache.lucene.store.IndexInput.readVLong(IndexInput.java:99)
at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:
134)
at org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache
$BloomFilterLoader.run(SimpleBloomCache.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server boss #2 ([id: 0x315cb235, /0.0.0.0:9200])" daemon
prio=10 tid=0x00002aaabddf9000 nid=0xe5b runnable [0x0000000042e56000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c5e31d0> (a sun.nio.ch.Util$2)
- locked <0x000000060c5e31e8> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c5e3260> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink
$Boss.run(NioServerSocketPipelineSink.java:240)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-6" daemon prio=10 tid=0x00002aaabd38a800
nid=0xe56 runnable [0x0000000042951000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c426e60> (a sun.nio.ch.Util$2)
- locked <0x000000060c426e78> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c426d40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-5" daemon prio=10 tid=0x00002aaabbe4c000
nid=0xe55 runnable [0x0000000042850000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c326830> (a sun.nio.ch.Util$2)
- locked <0x000000060c326848> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3263b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-1" daemon prio=10 tid=0x00002aaabd4d5000
nid=0xe54 runnable [0x000000004274f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3200f0> (a sun.nio.ch.Util$2)
- locked <0x000000060c320108> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322848> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-4" daemon prio=10 tid=0x00002aaabd294000
nid=0xe53 runnable [0x000000004264e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c323078> (a sun.nio.ch.Util$2)
- locked <0x000000060c323090> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322a40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-7" daemon prio=10 tid=0x00002aaabcc8d800
nid=0xe52 runnable [0x000000004254d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c320168> (a sun.nio.ch.Util$2)
- locked <0x000000060c320180> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3228d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-8" daemon prio=10 tid=0x00002aaabdc43800
nid=0xe51 runnable [0x000000004244c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3230f0> (a sun.nio.ch.Util$2)
- locked <0x000000060c323108> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322ad0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-3" daemon prio=10 tid=0x00002aaabcbce000
nid=0xe50 runnable [0x000000004234b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c324398> (a sun.nio.ch.Util$2)
- locked <0x000000060c324380> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322b60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-2" daemon prio=10 tid=0x00002aaabcfc0800
nid=0xe4f runnable [0x000000004224a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3237b8> (a sun.nio.ch.Util$2)
- locked <0x000000060c3237a0> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3229f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-8" daemon prio=10 tid=0x00002aaac056a800
nid=0xe4c runnable [0x0000000041f47000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3268d8> (a sun.nio.ch.Util$2)
- locked <0x000000060c3268f0> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3263f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-7" daemon prio=10 tid=0x00002aaac0350000
nid=0xe4b runnable [0x0000000041e46000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c326950> (a sun.nio.ch.Util$2)
- locked <0x000000060c326968> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c326488> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-6" daemon prio=10 tid=0x00002aaac034e800
nid=0xe4a runnable [0x0000000041d45000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c32a1a0> (a sun.nio.ch.Util$2)
- locked <0x000000060c32a188> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3267a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-5" daemon prio=10 tid=0x00002aaac04d5800
nid=0xe49 runnable [0x0000000041c44000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c37e728> (a sun.nio.ch.Util$2)
- locked <0x000000060c37e710> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c37e1b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-4" daemon prio=10 tid=0x00002aaac024d800
nid=0xe48 runnable [0x0000000041b43000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c328d70> (a sun.nio.ch.Util$2)
- locked <0x000000060c3284b8> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c326560> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-3" daemon prio=10 tid=0x00002aaac00a5800
nid=0xe47 runnable [0x0000000041a42000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c37ff08> (a sun.nio.ch.Util$2)
- locked <0x000000060c37fef0> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c37e248> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-2" daemon prio=10 tid=0x00002aaac0520000
nid=0xe46 runnable [0x0000000041941000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c380e78> (a sun.nio.ch.Util$2)
- locked <0x000000060c380e60> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c37e2d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-1" daemon prio=10 tid=0x00002aaabd13a000
nid=0xe45 runnable [0x0000000041840000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3d2408> (a sun.nio.ch.Util$2)
- locked <0x000000060c3d23f0> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c37e368> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server boss #1 ([id: 0x6fe22c75, /0.0.0.0:9300])" daemon
prio=10 tid=0x00002aaabce3b000 nid=0xe43 runnable [0x000000004143c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c328e90> (a sun.nio.ch.Util$2)
- locked <0x000000060c328e78> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3265f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink
$Boss.run(NioServerSocketPipelineSink.java:240)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"DestroyJavaVM" prio=10 tid=0x0000000040114800 nid=0xe11 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Wrapper-Connection" daemon prio=10 tid=0x00002aaabba54800 nid=0xe26
runnable [0x000000004133b000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.net.SocketInputStream.read(SocketInputStream.java:182)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at
org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:
4908)
at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:
5327)
at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=10 tid=0x00002aaabba23800 nid=0xe1e
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00002aaabba21800 nid=0xe1d
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00002aaabba1c800 nid=0xe1c
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00002aaabba1a000 nid=0xe1b
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00002aaabba18000
nid=0xe1a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"VM Thread" prio=10 tid=0x00002aaabb9ed800 nid=0xe17 runnable

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000040123000
nid=0xe12 runnable

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000040124800
nid=0xe13 runnable

"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000040126800
nid=0xe14 runnable

"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000040128800
nid=0xe15 runnable

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00002aaaafa79800
nid=0xe16 runnable

On Nov 21, 12:54 pm, Shay Banon kim...@gmail.com wrote:

Do you see any other thread busy?

p.s. Upgrade.

On Mon, Nov 21, 2011 at 5:40 PM, eroyal e...@spredfast.com wrote:

I am running Elasticsearch 0.15.2 on EC2 large instances. Every once
in a while a single CPU will run 100% for an extended period of time.
I cannot figure out what it is doing. At first I thought it might be
running a merge, but after dumping the stacktraces while it was
happening, the only merge related information I can find is this:

"elasticsearch[merge]-pool-6-thread-6602" daemon prio=10
tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition
[0x0000000044068000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x000000060c170fb8> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer
    $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:

at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
947)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

So it doesnt appear to be doing a merge. Any thoughts?

Thanks

4

I don't see anything evident as well. I suggest you upgrade and we can take
it from there.

On Tue, Nov 22, 2011 at 3:58 PM, eroyal egan@spredfast.com wrote:

Yeah there are several threads that are busy... I'd just like to
figure out which one(s) is running that much CPU for that long. I
dont 'think' it's a query as our performance logs dont show any query
taking that long. It also doesn't behave like a GC problem. These
are the runnable threads:

"Attach Listener" daemon prio=10 tid=0x00002aaae935e800 nid=0x6125
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"elasticsearch[cached]-pool-1-thread-31448" daemon prio=10
tid=0x00002aab1fbfd000 nid=0x5f19 runnable [0x000000004163e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610)
at org.apache.lucene.store.NIOFSDirectory
$NIOFSIndexInput.readInternal(NIOFSDirectory.java:168)
at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
160)
at

org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:
39)
at org.apache.lucene.store.IndexInput.readVLong(IndexInput.java:99)
at
org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:
134)
at org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache
$BloomFilterLoader.run(SimpleBloomCache.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server boss #2 ([id: 0x315cb235, /0.0.0.0:9200])" daemon
prio=10 tid=0x00002aaabddf9000 nid=0xe5b runnable [0x0000000042e56000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c5e31d0> (a sun.nio.ch.Util$2)
- locked <0x000000060c5e31e8> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c5e3260> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink
$Boss.run(NioServerSocketPipelineSink.java:240)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-6" daemon prio=10 tid=0x00002aaabd38a800
nid=0xe56 runnable [0x0000000042951000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c426e60> (a sun.nio.ch.Util$2)
- locked <0x000000060c426e78> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c426d40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-5" daemon prio=10 tid=0x00002aaabbe4c000
nid=0xe55 runnable [0x0000000042850000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c326830> (a sun.nio.ch.Util$2)
- locked <0x000000060c326848> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3263b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-1" daemon prio=10 tid=0x00002aaabd4d5000
nid=0xe54 runnable [0x000000004274f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3200f0> (a sun.nio.ch.Util$2)
- locked <0x000000060c320108> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322848> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-4" daemon prio=10 tid=0x00002aaabd294000
nid=0xe53 runnable [0x000000004264e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c323078> (a sun.nio.ch.Util$2)
- locked <0x000000060c323090> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322a40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-7" daemon prio=10 tid=0x00002aaabcc8d800
nid=0xe52 runnable [0x000000004254d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c320168> (a sun.nio.ch.Util$2)
- locked <0x000000060c320180> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3228d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-8" daemon prio=10 tid=0x00002aaabdc43800
nid=0xe51 runnable [0x000000004244c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3230f0> (a sun.nio.ch.Util$2)
- locked <0x000000060c323108> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322ad0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-3" daemon prio=10 tid=0x00002aaabcbce000
nid=0xe50 runnable [0x000000004234b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c324398> (a sun.nio.ch.Util$2)
- locked <0x000000060c324380> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322b60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-2" daemon prio=10 tid=0x00002aaabcfc0800
nid=0xe4f runnable [0x000000004224a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3237b8> (a sun.nio.ch.Util$2)
- locked <0x000000060c3237a0> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3229f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-8" daemon prio=10 tid=0x00002aaac056a800
nid=0xe4c runnable [0x0000000041f47000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3268d8> (a sun.nio.ch.Util$2)
- locked <0x000000060c3268f0> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3263f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-7" daemon prio=10 tid=0x00002aaac0350000
nid=0xe4b runnable [0x0000000041e46000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c326950> (a sun.nio.ch.Util$2)
- locked <0x000000060c326968> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c326488> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-6" daemon prio=10 tid=0x00002aaac034e800
nid=0xe4a runnable [0x0000000041d45000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c32a1a0> (a sun.nio.ch.Util$2)
- locked <0x000000060c32a188> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3267a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-5" daemon prio=10 tid=0x00002aaac04d5800
nid=0xe49 runnable [0x0000000041c44000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c37e728> (a sun.nio.ch.Util$2)
- locked <0x000000060c37e710> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c37e1b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-4" daemon prio=10 tid=0x00002aaac024d800
nid=0xe48 runnable [0x0000000041b43000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c328d70> (a sun.nio.ch.Util$2)
- locked <0x000000060c3284b8> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c326560> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-3" daemon prio=10 tid=0x00002aaac00a5800
nid=0xe47 runnable [0x0000000041a42000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c37ff08> (a sun.nio.ch.Util$2)
- locked <0x000000060c37fef0> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c37e248> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-2" daemon prio=10 tid=0x00002aaac0520000
nid=0xe46 runnable [0x0000000041941000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c380e78> (a sun.nio.ch.Util$2)
- locked <0x000000060c380e60> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c37e2d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server worker #1-1" daemon prio=10 tid=0x00002aaabd13a000
nid=0xe45 runnable [0x0000000041840000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3d2408> (a sun.nio.ch.Util$2)
- locked <0x000000060c3d23f0> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c37e368> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server boss #1 ([id: 0x6fe22c75, /0.0.0.0:9300])" daemon
prio=10 tid=0x00002aaabce3b000 nid=0xe43 runnable [0x000000004143c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c328e90> (a sun.nio.ch.Util$2)
- locked <0x000000060c328e78> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3265f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink
$Boss.run(NioServerSocketPipelineSink.java:240)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"DestroyJavaVM" prio=10 tid=0x0000000040114800 nid=0xe11 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Wrapper-Connection" daemon prio=10 tid=0x00002aaabba54800 nid=0xe26
runnable [0x000000004133b000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.net.SocketInputStream.read(SocketInputStream.java:182)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at
org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:
4908)
at
org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:
5327)
at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=10 tid=0x00002aaabba23800 nid=0xe1e
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00002aaabba21800 nid=0xe1d
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00002aaabba1c800 nid=0xe1c
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00002aaabba1a000 nid=0xe1b
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00002aaabba18000
nid=0xe1a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"VM Thread" prio=10 tid=0x00002aaabb9ed800 nid=0xe17 runnable

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000040123000
nid=0xe12 runnable

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000040124800
nid=0xe13 runnable

"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000040126800
nid=0xe14 runnable

"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000040128800
nid=0xe15 runnable

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00002aaaafa79800
nid=0xe16 runnable

On Nov 21, 12:54 pm, Shay Banon kim...@gmail.com wrote:

Do you see any other thread busy?

p.s. Upgrade.

On Mon, Nov 21, 2011 at 5:40 PM, eroyal e...@spredfast.com wrote:

I am running Elasticsearch 0.15.2 on EC2 large instances. Every once
in a while a single CPU will run 100% for an extended period of time.
I cannot figure out what it is doing. At first I thought it might be
running a merge, but after dumping the stacktraces while it was
happening, the only merge related information I can find is this:

"elasticsearch[merge]-pool-6-thread-6602" daemon prio=10
tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition
[0x0000000044068000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x000000060c170fb8> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer
    $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:

at

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:

at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

So it doesnt appear to be doing a merge. Any thoughts?

Thanks

5

Thanks for looking. For upgrading, do you recommend doing the upgrade
in place (of course in a test environment first) or doing a separate
install and migrating the data (~1 TB of data)? Thanks again for
looking... that is a big help.

On Nov 22, 11:56 am, Shay Banon kim...@gmail.com wrote:

I don't see anything evident as well. I suggest you upgrade and we can take
it from there.

On Tue, Nov 22, 2011 at 3:58 PM, eroyal e...@spredfast.com wrote:

Yeah there are several threads that are busy... I'd just like to
figure out which one(s) is running that much CPU for that long. I
dont 'think' it's a query as our performance logs dont show any query
taking that long. It also doesn't behave like a GC problem. These
are the runnable threads:

"Attach Listener" daemon prio=10 tid=0x00002aaae935e800 nid=0x6125
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"elasticsearch[cached]-pool-1-thread-31448" daemon prio=10
tid=0x00002aab1fbfd000 nid=0x5f19 runnable [0x000000004163e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610)
at org.apache.lucene.store.NIOFSDirectory
$NIOFSIndexInput.readInternal(NIOFSDirectory.java:168)
at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
160)
at

org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java :
39)
at org.apache.lucene.store.IndexInput.readVLong(IndexInput.java:99)
at
org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:
134)
at org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache
$BloomFilterLoader.run(SimpleBloomCache.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O server boss #2 ([id: 0x315cb235, /0.0.0.0:9200])" daemon
prio=10 tid=0x00002aaabddf9000 nid=0xe5b runnable [0x0000000042e56000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c5e31d0> (a sun.nio.ch.Util$2)
- locked <0x000000060c5e31e8> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c5e3260> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSi nk
$Boss.run(NioServerSocketPipelineSink.java:240)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-6" daemon prio=10 tid=0x00002aaabd38a800
nid=0xe56 runnable [0x0000000042951000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c426e60> (a sun.nio.ch.Util$2)
- locked <0x000000060c426e78> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c426d40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-5" daemon prio=10 tid=0x00002aaabbe4c000
nid=0xe55 runnable [0x0000000042850000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c326830> (a sun.nio.ch.Util$2)
- locked <0x000000060c326848> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3263b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-1" daemon prio=10 tid=0x00002aaabd4d5000
nid=0xe54 runnable [0x000000004274f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3200f0> (a sun.nio.ch.Util$2)
- locked <0x000000060c320108> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322848> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-4" daemon prio=10 tid=0x00002aaabd294000
nid=0xe53 runnable [0x000000004264e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c323078> (a sun.nio.ch.Util$2)
- locked <0x000000060c323090> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322a40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-7" daemon prio=10 tid=0x00002aaabcc8d800
nid=0xe52 runnable [0x000000004254d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c320168> (a sun.nio.ch.Util$2)
- locked <0x000000060c320180> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3228d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java:
38)
at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
163)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at

...

read more »

6

To be honest, I have not tested a direct upgrade from 0.15 to 0.18, I would
suggest testing it first. On production, if you can backup the data, it
would be great, though 1TB can be a large one. I suggest first test on a
test env, if direct upgrade does not work, upgrade in phases, to 0.16,
0.17, and then 0.18.

On Tue, Nov 22, 2011 at 11:05 PM, eroyal egan@spredfast.com wrote:

Thanks for looking. For upgrading, do you recommend doing the upgrade
in place (of course in a test environment first) or doing a separate
install and migrating the data (~1 TB of data)? Thanks again for
looking... that is a big help.

On Nov 22, 11:56 am, Shay Banon kim...@gmail.com wrote:

I don't see anything evident as well. I suggest you upgrade and we can
take
it from there.

On Tue, Nov 22, 2011 at 3:58 PM, eroyal e...@spredfast.com wrote:

Yeah there are several threads that are busy... I'd just like to
figure out which one(s) is running that much CPU for that long. I
dont 'think' it's a query as our performance logs dont show any query
taking that long. It also doesn't behave like a GC problem. These
are the runnable threads:

"Attach Listener" daemon prio=10 tid=0x00002aaae935e800 nid=0x6125
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"elasticsearch[cached]-pool-1-thread-31448" daemon prio=10
tid=0x00002aab1fbfd000 nid=0x5f19 runnable [0x000000004163e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610)
at org.apache.lucene.store.NIOFSDirectory
$NIOFSIndexInput.readInternal(NIOFSDirectory.java:168)
at

org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:

  1. at

org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java
:

  1. at
    org.apache.lucene.store.IndexInput.readVLong(IndexInput.java:99)
    at
    org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:
  2. at org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache
    $BloomFilterLoader.run(SimpleBloomCache.java:197)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

"New I/O server boss #2 ([id: 0x315cb235, /0.0.0.0:9200])" daemon
prio=10 tid=0x00002aaabddf9000 nid=0xe5b runnable [0x0000000042e56000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c5e31d0> (a sun.nio.ch.Util$2)
- locked <0x000000060c5e31e8> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c5e3260> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSi
nk

$Boss.run(NioServerSocketPipelineSink.java:240)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami
ngRunnable.java:

  1. at
    org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
    $1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-6" daemon prio=10 tid=0x00002aaabd38a800
nid=0xe56 runnable [0x0000000042951000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c426e60> (a sun.nio.ch.Util$2)
- locked <0x000000060c426e78> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c426d40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec
torUtil.java:

  1. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j
ava:

  1. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami
ngRunnable.java:

  1. at
    org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
    $1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-5" daemon prio=10 tid=0x00002aaabbe4c000
nid=0xe55 runnable [0x0000000042850000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c326830> (a sun.nio.ch.Util$2)
- locked <0x000000060c326848> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3263b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec
torUtil.java:

  1. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j
ava:

  1. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami
ngRunnable.java:

  1. at
    org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
    $1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-1" daemon prio=10 tid=0x00002aaabd4d5000
nid=0xe54 runnable [0x000000004274f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c3200f0> (a sun.nio.ch.Util$2)
- locked <0x000000060c320108> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322848> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec
torUtil.java:

  1. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j
ava:

  1. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami
ngRunnable.java:

  1. at
    org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
    $1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-4" daemon prio=10 tid=0x00002aaabd294000
nid=0xe53 runnable [0x000000004264e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c323078> (a sun.nio.ch.Util$2)
- locked <0x000000060c323090> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c322a40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec
torUtil.java:

  1. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j
ava:

  1. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami
ngRunnable.java:

  1. at
    org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
    $1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor
    $Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-7" daemon prio=10 tid=0x00002aaabcc8d800
nid=0xe52 runnable [0x000000004254d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000060c320168> (a sun.nio.ch.Util$2)
- locked <0x000000060c320180> (a java.util.Collections
$UnmodifiableSet)
- locked <0x000000060c3228d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at

org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec
torUtil.java:

  1. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j
ava:

  1. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami
ngRunnable.java:

  1. at
    org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
    $1.run(DeadLockProofWorker.java:44)
    at

...

read more »

7

Hello all,

just evaluating elasticsearch for our environment and had the same
issue.
I'm running es 0.18.4 with 3 nodes (5 shards and 3 replicas).

Its running great. But sometimes one CPU hangs at 100%. The node
doesn't response anymore.

Any solutions out there since the problem got mentioned in this thread
a while ago?
If not, any hints where to look at? What do you need for further
investigations?

Thx in advance and greetings from germany
maik

On 21 Nov., 16:40, eroyal e...@spredfast.com wrote:

I am running Elasticsearch 0.15.2 on EC2 large instances. Every once
in a while a singleCPUwill run 100% for an extended period of time.
I cannot figure out what it is doing. At first I thought it might be
running a merge, but after dumping the stacktraces while it was
happening, the only merge related information I can find is this:

"elasticsearch[merge]-pool-6-thread-6602" daemon prio=10
tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition
[0x0000000044068000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x000000060c170fb8> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer
    $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:

at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
947)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

So it doesnt appear to be doing a merge. Any thoughts?

Thanks

8

This problem doesn't occure the whole last week.
Just testing stability with stresstests and everything works fine so far.

Greetings
maik

9

Has anything changed? One reason for the high CPU usage might be a big in
the networking layer (netty) that is fixed in 0.18.5 (which includes a
newer version

On Thu, Dec 8, 2011 at 9:50 AM, maik maik2102@googlemail.com wrote:

This problem doesn't occure the whole last week.
Just testing stability with stresstests and everything works fine so far.

Greetings
maik

10

I changed the mappings and the client-side handling, but the server
configuration did not change.
Dont know what this could be about..