# Every once in a while a single CPU will run 100% for an extended period of time **URL:** \ **Category:** Elasticsearch\ **Created:** [November 21, 2011, 3:40pm UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925 "2011-11-21T15:40:50Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![eroyal](https://avatars.discourse-cdn.com/v4/letter/e/bb73d2/32.png) [@eroyal](https://discuss.elastic.co/u/eroyal)\ **Post date:** [November 21, 2011, 3:40pm UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/1 "2011-11-21T15:40:50Z")
I am running Elasticsearch 0.15.2 on EC2 large instances. Every once in a while a single CPU will run 100% for an extended period of time. I cannot figure out what it is doing. At first I thought it might be running a merge, but after dumping the stacktraces while it was happening, the only merge related information I can find is this: "elasticsearch[merge]-pool-6-thread-6602" daemon prio=10 tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition [0x0000000044068000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for \<0x000000060c170fb8\> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer $ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java: 1. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java: 947) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) So it doesnt appear to be doing a merge. Any thoughts? Thanks ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 21, 2011, 6:54pm UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/2 "2011-11-21T18:54:45Z")
Do you see any other thread busy? p.s. Upgrade. On Mon, Nov 21, 2011 at 5:40 PM, eroyal [egan@spredfast.com](mailto:egan@spredfast.com) wrote: > I am running Elasticsearch 0.15.2 on EC2 large instances. Every once > in a while a single CPU will run 100% for an extended period of time. > I cannot figure out what it is doing. At first I thought it might be > running a merge, but after dumping the stacktraces while it was > happening, the only merge related information I can find is this: > > "elasticsearch[merge]-pool-6-thread-6602" daemon prio=10 > tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition > [0x0000000044068000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for \<0x000000060c170fb8\> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at java.util.concurrent.locks.AbstractQueuedSynchronizer > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java: > > 1. > > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java: > 947) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:662) > > So it doesnt appear to be doing a merge. Any thoughts? > > Thanks ---
**Author:** ![eroyal](https://avatars.discourse-cdn.com/v4/letter/e/bb73d2/32.png) [@eroyal](https://discuss.elastic.co/u/eroyal)\ **Post date:** [November 22, 2011, 1:58pm UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/3 "2011-11-22T13:58:08Z")
Yeah there are several threads that are busy... I'd just like to figure out which one(s) is running that much CPU for that long. I dont 'think' it's a query as our performance logs dont show any query taking that long. It also doesn't behave like a GC problem. These are the runnable threads: "Attach Listener" daemon prio=10 tid=0x00002aaae935e800 nid=0x6125 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "elasticsearch[cached]-pool-1-thread-31448" daemon prio=10 tid=0x00002aab1fbfd000 nid=0x5f19 runnable [0x000000004163e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.NativeThread.current(Native Method) at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610) at org.apache.lucene.store.NIOFSDirectory $NIOFSIndexInput.readInternal(NIOFSDirectory.java:168) at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java: 160) at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java: 39) at org.apache.lucene.store.IndexInput.readVLong(IndexInput.java:99) at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java: 134) at org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache $BloomFilterLoader.run(SimpleBloomCache.java:197) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server boss #2 ([id: 0x315cb235, /0.0.0.0:9200])" daemon prio=10 tid=0x00002aaabddf9000 nid=0xe5b runnable [0x0000000042e56000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c5e31d0\> (a sun.nio.ch.Util$2) - locked \<0x000000060c5e31e8\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c5e3260\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink $Boss.run(NioServerSocketPipelineSink.java:240) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O client worker #1-6" daemon prio=10 tid=0x00002aaabd38a800 nid=0xe56 runnable [0x0000000042951000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c426e60\> (a sun.nio.ch.Util$2) - locked \<0x000000060c426e78\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c426d40\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O client worker #1-5" daemon prio=10 tid=0x00002aaabbe4c000 nid=0xe55 runnable [0x0000000042850000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c326830\> (a sun.nio.ch.Util$2) - locked \<0x000000060c326848\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c3263b0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O client worker #1-1" daemon prio=10 tid=0x00002aaabd4d5000 nid=0xe54 runnable [0x000000004274f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c3200f0\> (a sun.nio.ch.Util$2) - locked \<0x000000060c320108\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c322848\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O client worker #1-4" daemon prio=10 tid=0x00002aaabd294000 nid=0xe53 runnable [0x000000004264e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c323078\> (a sun.nio.ch.Util$2) - locked \<0x000000060c323090\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c322a40\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O client worker #1-7" daemon prio=10 tid=0x00002aaabcc8d800 nid=0xe52 runnable [0x000000004254d000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c320168\> (a sun.nio.ch.Util$2) - locked \<0x000000060c320180\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c3228d8\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O client worker #1-8" daemon prio=10 tid=0x00002aaabdc43800 nid=0xe51 runnable [0x000000004244c000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c3230f0\> (a sun.nio.ch.Util$2) - locked \<0x000000060c323108\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c322ad0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O client worker #1-3" daemon prio=10 tid=0x00002aaabcbce000 nid=0xe50 runnable [0x000000004234b000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c324398\> (a sun.nio.ch.Util$2) - locked \<0x000000060c324380\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c322b60\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O client worker #1-2" daemon prio=10 tid=0x00002aaabcfc0800 nid=0xe4f runnable [0x000000004224a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c3237b8\> (a sun.nio.ch.Util$2) - locked \<0x000000060c3237a0\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c3229f8\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server worker #1-8" daemon prio=10 tid=0x00002aaac056a800 nid=0xe4c runnable [0x0000000041f47000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c3268d8\> (a sun.nio.ch.Util$2) - locked \<0x000000060c3268f0\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c3263f8\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server worker #1-7" daemon prio=10 tid=0x00002aaac0350000 nid=0xe4b runnable [0x0000000041e46000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c326950\> (a sun.nio.ch.Util$2) - locked \<0x000000060c326968\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c326488\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server worker #1-6" daemon prio=10 tid=0x00002aaac034e800 nid=0xe4a runnable [0x0000000041d45000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c32a1a0\> (a sun.nio.ch.Util$2) - locked \<0x000000060c32a188\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c3267a0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server worker #1-5" daemon prio=10 tid=0x00002aaac04d5800 nid=0xe49 runnable [0x0000000041c44000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c37e728\> (a sun.nio.ch.Util$2) - locked \<0x000000060c37e710\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c37e1b8\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server worker #1-4" daemon prio=10 tid=0x00002aaac024d800 nid=0xe48 runnable [0x0000000041b43000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c328d70\> (a sun.nio.ch.Util$2) - locked \<0x000000060c3284b8\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c326560\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server worker #1-3" daemon prio=10 tid=0x00002aaac00a5800 nid=0xe47 runnable [0x0000000041a42000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c37ff08\> (a sun.nio.ch.Util$2) - locked \<0x000000060c37fef0\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c37e248\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server worker #1-2" daemon prio=10 tid=0x00002aaac0520000 nid=0xe46 runnable [0x0000000041941000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c380e78\> (a sun.nio.ch.Util$2) - locked \<0x000000060c380e60\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c37e2d8\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server worker #1-1" daemon prio=10 tid=0x00002aaabd13a000 nid=0xe45 runnable [0x0000000041840000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c3d2408\> (a sun.nio.ch.Util$2) - locked \<0x000000060c3d23f0\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c37e368\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: 38) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 163) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "New I/O server boss #1 ([id: 0x6fe22c75, /0.0.0.0:9300])" daemon prio=10 tid=0x00002aaabce3b000 nid=0xe43 runnable [0x000000004143c000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked \<0x000000060c328e90\> (a sun.nio.ch.Util$2) - locked \<0x000000060c328e78\> (a java.util.Collections $UnmodifiableSet) - locked \<0x000000060c3265f0\> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink $Boss.run(NioServerSocketPipelineSink.java:240) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "DestroyJavaVM" prio=10 tid=0x0000000040114800 nid=0xe11 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Wrapper-Connection" daemon prio=10 tid=0x00002aaabba54800 nid=0xe26 runnable [0x000000004133b000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.net.SocketInputStream.read(SocketInputStream.java:182) at java.io.DataInputStream.readByte(DataInputStream.java:248) at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java: 4908) at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java: 5327) at java.lang.Thread.run(Thread.java:662) "Low Memory Detector" daemon prio=10 tid=0x00002aaabba23800 nid=0xe1e runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "CompilerThread1" daemon prio=10 tid=0x00002aaabba21800 nid=0xe1d waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "CompilerThread0" daemon prio=10 tid=0x00002aaabba1c800 nid=0xe1c waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00002aaabba1a000 nid=0xe1b runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00002aaabba18000 nid=0xe1a waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "VM Thread" prio=10 tid=0x00002aaabb9ed800 nid=0xe17 runnable "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000040123000 nid=0xe12 runnable "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000040124800 nid=0xe13 runnable "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000040126800 nid=0xe14 runnable "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000040128800 nid=0xe15 runnable "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00002aaaafa79800 nid=0xe16 runnable On Nov 21, 12:54 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Do you see any other thread busy? > > p.s. Upgrade. > > On Mon, Nov 21, 2011 at 5:40 PM, eroyal [e...@spredfast.com](mailto:e...@spredfast.com) wrote: > > > I am running Elasticsearch 0.15.2 on EC2 large instances. Every once > > in a while a single CPU will run 100% for an extended period of time. > > I cannot figure out what it is doing. At first I thought it might be > > running a merge, but after dumping the stacktraces while it was > > happening, the only merge related information I can find is this: > > > "elasticsearch[merge]-pool-6-thread-6602" daemon prio=10 > > tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition > > [0x0000000044068000] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > > > - parking to wait for \<0x000000060c170fb8\> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at java.util.concurrent.locks.AbstractQueuedSynchronizer > > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > at > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java: > > > > 1. > > > > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java: > > 947) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:907) > > at java.lang.Thread.run(Thread.java:662) > > > So it doesnt appear to be doing a merge. Any thoughts? > > > Thanks ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 22, 2011, 5:56pm UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/4 "2011-11-22T17:56:49Z")
I don't see anything evident as well. I suggest you upgrade and we can take it from there. On Tue, Nov 22, 2011 at 3:58 PM, eroyal [egan@spredfast.com](mailto:egan@spredfast.com) wrote: > Yeah there are several threads that are busy... I'd just like to > figure out which one(s) is running that much CPU for that long. I > dont 'think' it's a query as our performance logs dont show any query > taking that long. It also doesn't behave like a GC problem. These > are the runnable threads: > > "Attach Listener" daemon prio=10 tid=0x00002aaae935e800 nid=0x6125 > runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "elasticsearch[cached]-pool-1-thread-31448" daemon prio=10 > tid=0x00002aab1fbfd000 nid=0x5f19 runnable [0x000000004163e000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.NativeThread.current(Native Method) > at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27) > at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610) > at org.apache.lucene.store.NIOFSDirectory > $NIOFSIndexInput.readInternal(NIOFSDirectory.java:168) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java: > 160) > at > > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java: > 39) > at org.apache.lucene.store.IndexInput.readVLong(IndexInput.java:99) > at > org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java: > 134) > at org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache > $BloomFilterLoader.run(SimpleBloomCache.java:197) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server boss #2 ([id: 0x315cb235, /0.0.0.0:9200])" daemon > prio=10 tid=0x00002aaabddf9000 nid=0xe5b runnable [0x0000000042e56000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c5e31d0\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c5e31e8\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c5e3260\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink > $Boss.run(NioServerSocketPipelineSink.java:240) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O client worker #1-6" daemon prio=10 tid=0x00002aaabd38a800 > nid=0xe56 runnable [0x0000000042951000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c426e60\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c426e78\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c426d40\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O client worker #1-5" daemon prio=10 tid=0x00002aaabbe4c000 > nid=0xe55 runnable [0x0000000042850000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c326830\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c326848\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c3263b0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O client worker #1-1" daemon prio=10 tid=0x00002aaabd4d5000 > nid=0xe54 runnable [0x000000004274f000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c3200f0\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c320108\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c322848\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O client worker #1-4" daemon prio=10 tid=0x00002aaabd294000 > nid=0xe53 runnable [0x000000004264e000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c323078\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c323090\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c322a40\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O client worker #1-7" daemon prio=10 tid=0x00002aaabcc8d800 > nid=0xe52 runnable [0x000000004254d000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c320168\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c320180\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c3228d8\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O client worker #1-8" daemon prio=10 tid=0x00002aaabdc43800 > nid=0xe51 runnable [0x000000004244c000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c3230f0\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c323108\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c322ad0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O client worker #1-3" daemon prio=10 tid=0x00002aaabcbce000 > nid=0xe50 runnable [0x000000004234b000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c324398\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c324380\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c322b60\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O client worker #1-2" daemon prio=10 tid=0x00002aaabcfc0800 > nid=0xe4f runnable [0x000000004224a000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c3237b8\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c3237a0\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c3229f8\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server worker #1-8" daemon prio=10 tid=0x00002aaac056a800 > nid=0xe4c runnable [0x0000000041f47000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c3268d8\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c3268f0\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c3263f8\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server worker #1-7" daemon prio=10 tid=0x00002aaac0350000 > nid=0xe4b runnable [0x0000000041e46000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c326950\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c326968\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c326488\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server worker #1-6" daemon prio=10 tid=0x00002aaac034e800 > nid=0xe4a runnable [0x0000000041d45000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c32a1a0\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c32a188\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c3267a0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server worker #1-5" daemon prio=10 tid=0x00002aaac04d5800 > nid=0xe49 runnable [0x0000000041c44000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c37e728\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c37e710\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c37e1b8\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server worker #1-4" daemon prio=10 tid=0x00002aaac024d800 > nid=0xe48 runnable [0x0000000041b43000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c328d70\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c3284b8\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c326560\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server worker #1-3" daemon prio=10 tid=0x00002aaac00a5800 > nid=0xe47 runnable [0x0000000041a42000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c37ff08\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c37fef0\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c37e248\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server worker #1-2" daemon prio=10 tid=0x00002aaac0520000 > nid=0xe46 runnable [0x0000000041941000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c380e78\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c380e60\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c37e2d8\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server worker #1-1" daemon prio=10 tid=0x00002aaabd13a000 > nid=0xe45 runnable [0x0000000041840000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c3d2408\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c3d23f0\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c37e368\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java: > 38) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 163) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "New I/O server boss #1 ([id: 0x6fe22c75, /0.0.0.0:9300])" daemon > prio=10 tid=0x00002aaabce3b000 nid=0xe43 runnable [0x000000004143c000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked \<0x000000060c328e90\> (a sun.nio.ch.Util$2) > - locked \<0x000000060c328e78\> (a java.util.Collections > $UnmodifiableSet) > - locked \<0x000000060c3265f0\> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink > $Boss.run(NioServerSocketPipelineSink.java:240) > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > "DestroyJavaVM" prio=10 tid=0x0000000040114800 nid=0xe11 waiting on > condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "Wrapper-Connection" daemon prio=10 tid=0x00002aaabba54800 nid=0xe26 > runnable [0x000000004133b000] > java.lang.Thread.State: RUNNABLE > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:129) > at java.net.SocketInputStream.read(SocketInputStream.java:182) > at java.io.DataInputStream.readByte(DataInputStream.java:248) > at > org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java: > 4908) > at > org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java: > 5327) > at java.lang.Thread.run(Thread.java:662) > > "Low Memory Detector" daemon prio=10 tid=0x00002aaabba23800 nid=0xe1e > runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "CompilerThread1" daemon prio=10 tid=0x00002aaabba21800 nid=0xe1d > waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "CompilerThread0" daemon prio=10 tid=0x00002aaabba1c800 nid=0xe1c > waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "Signal Dispatcher" daemon prio=10 tid=0x00002aaabba1a000 nid=0xe1b > runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00002aaabba18000 > nid=0xe1a waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "VM Thread" prio=10 tid=0x00002aaabb9ed800 nid=0xe17 runnable > > "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000040123000 > nid=0xe12 runnable > > "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000040124800 > nid=0xe13 runnable > > "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000040126800 > nid=0xe14 runnable > > "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000040128800 > nid=0xe15 runnable > > "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00002aaaafa79800 > nid=0xe16 runnable > > On Nov 21, 12:54 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Do you see any other thread busy? > > > > p.s. Upgrade. > > > > On Mon, Nov 21, 2011 at 5:40 PM, eroyal [e...@spredfast.com](mailto:e...@spredfast.com) wrote: > > > > > I am running Elasticsearch 0.15.2 on EC2 large instances. Every once > > > in a while a single CPU will run 100% for an extended period of time. > > > I cannot figure out what it is doing. At first I thought it might be > > > running a merge, but after dumping the stacktraces while it was > > > happening, the only merge related information I can find is this: > > > > > "elasticsearch[merge]-pool-6-thread-6602" daemon prio=10 > > > tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition > > > [0x0000000044068000] > > > java.lang.Thread.State: WAITING (parking) > > > at sun.misc.Unsafe.park(Native Method) > > > > > > - parking to wait for \<0x000000060c170fb8\> (a > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer > > > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > > at > > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java: > > > > > > 1. > > > > > > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java: > > > > 1. > > > > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:907) > > > at java.lang.Thread.run(Thread.java:662) > > > > > So it doesnt appear to be doing a merge. Any thoughts? > > > > > Thanks ---
**Author:** ![eroyal](https://avatars.discourse-cdn.com/v4/letter/e/bb73d2/32.png) [@eroyal](https://discuss.elastic.co/u/eroyal)\ **Post date:** [November 22, 2011, 9:05pm UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/5 "2011-11-22T21:05:20Z")
Thanks for looking. For upgrading, do you recommend doing the upgrade in place (of course in a test environment first) or doing a separate install and migrating the data (~1 TB of data)? Thanks again for looking... that is a big help. On Nov 22, 11:56 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > I don't see anything evident as well. I suggest you upgrade and we can take > it from there. > > On Tue, Nov 22, 2011 at 3:58 PM, eroyal [e...@spredfast.com](mailto:e...@spredfast.com) wrote: > > > Yeah there are several threads that are busy... I'd just like to > > figure out which one(s) is running that much CPU for that long. I > > dont 'think' it's a query as our performance logs dont show any query > > taking that long. It also doesn't behave like a GC problem. These > > are the runnable threads: > > > "Attach Listener" daemon prio=10 tid=0x00002aaae935e800 nid=0x6125 > > runnable [0x0000000000000000] > > java.lang.Thread.State: RUNNABLE > > > "elasticsearch[cached]-pool-1-thread-31448" daemon prio=10 > > tid=0x00002aab1fbfd000 nid=0x5f19 runnable [0x000000004163e000] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.NativeThread.current(Native Method) > > at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27) > > at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610) > > at org.apache.lucene.store.NIOFSDirectory > > $NIOFSIndexInput.readInternal(NIOFSDirectory.java:168) > > at > > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java: > > 160) > > at > > > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java : > > 39) > > at org.apache.lucene.store.IndexInput.readVLong(IndexInput.java:99) > > at > > org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java: > > 134) > > at org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache > > $BloomFilterLoader.run(SimpleBloomCache.java:197) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > "New I/O server boss #2 ([id: 0x315cb235, /0.0.0.0:9200])" daemon > > prio=10 tid=0x00002aaabddf9000 nid=0xe5b runnable [0x0000000042e56000] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > - locked \<0x000000060c5e31d0\> (a sun.nio.ch.Util$2) > > - locked \<0x000000060c5e31e8\> (a java.util.Collections > > $UnmodifiableSet) > > - locked \<0x000000060c5e3260\> (a sun.nio.ch.EPollSelectorImpl) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSi nk > > $Boss.run(NioServerSocketPipelineSink.java:240) > > at > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java: > > 108) > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > $1.run(DeadLockProofWorker.java:44) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > "New I/O client worker #1-6" daemon prio=10 tid=0x00002aaabd38a800 > > nid=0xe56 runnable [0x0000000042951000] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > - locked \<0x000000060c426e60\> (a sun.nio.ch.Util$2) > > - locked \<0x000000060c426e78\> (a java.util.Collections > > $UnmodifiableSet) > > - locked \<0x000000060c426d40\> (a sun.nio.ch.EPollSelectorImpl) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java: > > 38) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava: > > 163) > > at > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java: > > 108) > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > $1.run(DeadLockProofWorker.java:44) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > "New I/O client worker #1-5" daemon prio=10 tid=0x00002aaabbe4c000 > > nid=0xe55 runnable [0x0000000042850000] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > - locked \<0x000000060c326830\> (a sun.nio.ch.Util$2) > > - locked \<0x000000060c326848\> (a java.util.Collections > > $UnmodifiableSet) > > - locked \<0x000000060c3263b0\> (a sun.nio.ch.EPollSelectorImpl) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java: > > 38) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava: > > 163) > > at > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java: > > 108) > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > $1.run(DeadLockProofWorker.java:44) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > "New I/O client worker #1-1" daemon prio=10 tid=0x00002aaabd4d5000 > > nid=0xe54 runnable [0x000000004274f000] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > - locked \<0x000000060c3200f0\> (a sun.nio.ch.Util$2) > > - locked \<0x000000060c320108\> (a java.util.Collections > > $UnmodifiableSet) > > - locked \<0x000000060c322848\> (a sun.nio.ch.EPollSelectorImpl) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java: > > 38) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava: > > 163) > > at > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java: > > 108) > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > $1.run(DeadLockProofWorker.java:44) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > "New I/O client worker #1-4" daemon prio=10 tid=0x00002aaabd294000 > > nid=0xe53 runnable [0x000000004264e000] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > - locked \<0x000000060c323078\> (a sun.nio.ch.Util$2) > > - locked \<0x000000060c323090\> (a java.util.Collections > > $UnmodifiableSet) > > - locked \<0x000000060c322a40\> (a sun.nio.ch.EPollSelectorImpl) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java: > > 38) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava: > > 163) > > at > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java: > > 108) > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > $1.run(DeadLockProofWorker.java:44) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > "New I/O client worker #1-7" daemon prio=10 tid=0x00002aaabcc8d800 > > nid=0xe52 runnable [0x000000004254d000] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > - locked \<0x000000060c320168\> (a sun.nio.ch.Util$2) > > - locked \<0x000000060c320180\> (a java.util.Collections > > $UnmodifiableSet) > > - locked \<0x000000060c3228d8\> (a sun.nio.ch.EPollSelectorImpl) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec torUtil.java: > > 38) > > at > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava: > > 163) > > at > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java: > > 108) > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > $1.run(DeadLockProofWorker.java:44) > > at > > ... > > read more » ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 23, 2011, 10:22am UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/6 "2011-11-23T10:22:56Z")
To be honest, I have not tested a direct upgrade from 0.15 to 0.18, I would suggest testing it first. On production, if you can backup the data, it would be great, though 1TB can be a large one. I suggest first test on a test env, if direct upgrade does not work, upgrade in phases, to 0.16, 0.17, and then 0.18. On Tue, Nov 22, 2011 at 11:05 PM, eroyal [egan@spredfast.com](mailto:egan@spredfast.com) wrote: > Thanks for looking. For upgrading, do you recommend doing the upgrade > in place (of course in a test environment first) or doing a separate > install and migrating the data (~1 TB of data)? Thanks again for > looking... that is a big help. > > On Nov 22, 11:56 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > I don't see anything evident as well. I suggest you upgrade and we can > > take > > it from there. > > > > On Tue, Nov 22, 2011 at 3:58 PM, eroyal [e...@spredfast.com](mailto:e...@spredfast.com) wrote: > > > > > Yeah there are several threads that are busy... I'd just like to > > > figure out which one(s) is running that much CPU for that long. I > > > dont 'think' it's a query as our performance logs dont show any query > > > taking that long. It also doesn't behave like a GC problem. These > > > are the runnable threads: > > > > > "Attach Listener" daemon prio=10 tid=0x00002aaae935e800 nid=0x6125 > > > runnable [0x0000000000000000] > > > java.lang.Thread.State: RUNNABLE > > > > > "elasticsearch[cached]-pool-1-thread-31448" daemon prio=10 > > > tid=0x00002aab1fbfd000 nid=0x5f19 runnable [0x000000004163e000] > > > java.lang.Thread.State: RUNNABLE > > > at sun.nio.ch.NativeThread.current(Native Method) > > > at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27) > > > at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610) > > > at org.apache.lucene.store.NIOFSDirectory > > > $NIOFSIndexInput.readInternal(NIOFSDirectory.java:168) > > > at > > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java: > > > > 1. at > > > > > > > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java > : > > > > 1. at > > > org.apache.lucene.store.IndexInput.readVLong(IndexInput.java:99) > > > at > > > org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java: > > > 2. at org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache > > > $BloomFilterLoader.run(SimpleBloomCache.java:197) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:662) > > > > > "New I/O server boss #2 ([id: 0x315cb235, /0.0.0.0:9200])" daemon > > > prio=10 tid=0x00002aaabddf9000 nid=0xe5b runnable [0x0000000042e56000] > > > java.lang.Thread.State: RUNNABLE > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > > - locked \<0x000000060c5e31d0\> (a sun.nio.ch.Util$2) > > > - locked \<0x000000060c5e31e8\> (a java.util.Collections > > > $UnmodifiableSet) > > > - locked \<0x000000060c5e3260\> (a sun.nio.ch.EPollSelectorImpl) > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > > at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSi > nk > > > > $Boss.run(NioServerSocketPipelineSink.java:240) > > > at > > > > > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami > ngRunnable.java: > > > > 1. at > > > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > > $1.run(DeadLockProofWorker.java:44) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:662) > > > > > "New I/O client worker #1-6" daemon prio=10 tid=0x00002aaabd38a800 > > > nid=0xe56 runnable [0x0000000042951000] > > > java.lang.Thread.State: RUNNABLE > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > > - locked \<0x000000060c426e60\> (a sun.nio.ch.Util$2) > > > - locked \<0x000000060c426e78\> (a java.util.Collections > > > $UnmodifiableSet) > > > - locked \<0x000000060c426d40\> (a sun.nio.ch.EPollSelectorImpl) > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > > at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec > torUtil.java: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j > ava: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami > ngRunnable.java: > > > > 1. at > > > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > > $1.run(DeadLockProofWorker.java:44) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:662) > > > > > "New I/O client worker #1-5" daemon prio=10 tid=0x00002aaabbe4c000 > > > nid=0xe55 runnable [0x0000000042850000] > > > java.lang.Thread.State: RUNNABLE > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > > - locked \<0x000000060c326830\> (a sun.nio.ch.Util$2) > > > - locked \<0x000000060c326848\> (a java.util.Collections > > > $UnmodifiableSet) > > > - locked \<0x000000060c3263b0\> (a sun.nio.ch.EPollSelectorImpl) > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > > at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec > torUtil.java: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j > ava: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami > ngRunnable.java: > > > > 1. at > > > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > > $1.run(DeadLockProofWorker.java:44) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:662) > > > > > "New I/O client worker #1-1" daemon prio=10 tid=0x00002aaabd4d5000 > > > nid=0xe54 runnable [0x000000004274f000] > > > java.lang.Thread.State: RUNNABLE > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > > - locked \<0x000000060c3200f0\> (a sun.nio.ch.Util$2) > > > - locked \<0x000000060c320108\> (a java.util.Collections > > > $UnmodifiableSet) > > > - locked \<0x000000060c322848\> (a sun.nio.ch.EPollSelectorImpl) > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > > at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec > torUtil.java: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j > ava: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami > ngRunnable.java: > > > > 1. at > > > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > > $1.run(DeadLockProofWorker.java:44) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:662) > > > > > "New I/O client worker #1-4" daemon prio=10 tid=0x00002aaabd294000 > > > nid=0xe53 runnable [0x000000004264e000] > > > java.lang.Thread.State: RUNNABLE > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > > - locked \<0x000000060c323078\> (a sun.nio.ch.Util$2) > > > - locked \<0x000000060c323090\> (a java.util.Collections > > > $UnmodifiableSet) > > > - locked \<0x000000060c322a40\> (a sun.nio.ch.EPollSelectorImpl) > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > > at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec > torUtil.java: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j > ava: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami > ngRunnable.java: > > > > 1. at > > > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > > $1.run(DeadLockProofWorker.java:44) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:662) > > > > > "New I/O client worker #1-7" daemon prio=10 tid=0x00002aaabcc8d800 > > > nid=0xe52 runnable [0x000000004254d000] > > > java.lang.Thread.State: RUNNABLE > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > > - locked \<0x000000060c320168\> (a sun.nio.ch.Util$2) > > > - locked \<0x000000060c320180\> (a java.util.Collections > > > $UnmodifiableSet) > > > - locked \<0x000000060c3228d8\> (a sun.nio.ch.EPollSelectorImpl) > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > > at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(Selec > torUtil.java: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j > ava: > > > > 1. at > > > > > > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami > ngRunnable.java: > > > > 1. at > > > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > > $1.run(DeadLockProofWorker.java:44) > > > at > > > > ... > > > > read more » ---
**Author:** ![maik](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@maik](https://discuss.elastic.co/u/maik)\ **Post date:** [November 29, 2011, 8:14am UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/7 "2011-11-29T08:14:45Z")
Hello all, just evaluating elasticsearch for our environment and had the same issue. I'm running es 0.18.4 with 3 nodes (5 shards and 3 replicas). Its running great. But sometimes one CPU hangs at 100%. The node doesn't response anymore. Any solutions out there since the problem got mentioned in this thread a while ago? If not, any hints where to look at? What do you need for further investigations? Thx in advance and greetings from germany maik On 21 Nov., 16:40, eroyal [e...@spredfast.com](mailto:e...@spredfast.com) wrote: > I am running Elasticsearch 0.15.2 on EC2 large instances. Every once > in a while a singleCPUwill run 100% for an extended period of time. > I cannot figure out what it is doing. At first I thought it might be > running a merge, but after dumping the stacktraces while it was > happening, the only merge related information I can find is this: > > "elasticsearch[merge]-pool-6-thread-6602" daemon prio=10 > tid=0x00002aab1fc7b800 nid=0x60ed waiting on condition > [0x0000000044068000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for \<0x000000060c170fb8\> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at java.util.concurrent.locks.AbstractQueuedSynchronizer > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java: > > 1. > > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java: > 947) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:662) > > So it doesnt appear to be doing a merge. Any thoughts? > > Thanks ---
**Author:** ![maik](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@maik](https://discuss.elastic.co/u/maik)\ **Post date:** [December 8, 2011, 7:50am UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/8 "2011-12-08T07:50:43Z")
This problem doesn't occure the whole last week. Just testing stability with stresstests and everything works fine so far. Greetings maik ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 8, 2011, 4:24pm UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/9 "2011-12-08T16:24:11Z")
Has anything changed? One reason for the high CPU usage might be a big in the networking layer (netty) that is fixed in 0.18.5 (which includes a newer version On Thu, Dec 8, 2011 at 9:50 AM, maik [maik2102@googlemail.com](mailto:maik2102@googlemail.com) wrote: > This problem doesn't occure the whole last week. > Just testing stability with stresstests and everything works fine so far. > > Greetings > maik ---
**Author:** ![maik](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@maik](https://discuss.elastic.co/u/maik)\ **Post date:** [December 9, 2011, 9:19am UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/10 "2011-12-09T09:19:08Z")
I changed the mappings and the client-side handling, but the server configuration did not change. Dont know what this could be about.. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:45am UTC](https://discuss.elastic.co/t/every-once-in-a-while-a-single-cpu-will-run-100-for-an-extended-period-of-time/5925/11 "2017-07-06T03:45:58Z")