# Lock up on actionGet **URL:** \ **Category:** Elasticsearch\ **Created:** [February 12, 2012, 12:38am UTC](https://discuss.elastic.co/t/lock-up-on-actionget/6678 "2012-02-12T00:38:34Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Frank\_LaRosa](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@Frank\_LaRosa](https://discuss.elastic.co/u/Frank_LaRosa)\ **Post date:** [February 12, 2012, 12:38am UTC](https://discuss.elastic.co/t/lock-up-on-actionget/6678/1 "2012-02-12T00:38:34Z")
Hi, I've been doing a large number of updates and queries to my index over the past couple of days. It currently contains about 10 million documents which is about half the data I intend to insert. Twice today, my job locked up waiting on the result of a query. I had to kill the java process and restart it. I have since started using actionGet with the timeout parameter, but I'm worried as to what the underlying cause of the lockup might be. It only happened twice over many millions of operations. I obtained this stack trace the last time it happened, not sure if has any helpful information: "Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on condition [0x000000004446c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for \<0x00000000982e6728\> (a org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java: 811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java: 969) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java: 1281) at org.elasticsearch.common.util.concurrent.AbstractFuture $Sync.get(AbstractFuture.java:249) at org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture.java: 78) at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java: 44) at com.studyblue.managers.impl.IdeaGraphManagerImpl $GetIdeaThread.run(IdeaGraphManagerImpl.java:451) I have the stack traces from the rest of the threads if that would help. Thanks. Frank ---
**Author:** ![Frank\_LaRosa](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@Frank\_LaRosa](https://discuss.elastic.co/u/Frank_LaRosa)\ **Post date:** [February 12, 2012, 4:01am UTC](https://discuss.elastic.co/t/lock-up-on-actionget/6678/2 "2012-02-12T04:01:10Z")
After adding a timeout, I'm now seeing many hundreds of "ElasticSearchTimeoutException". I've stopped and restarted my application, but the error persists. Any idea what's going on? On Feb 11, 6:38 pm, Frank LaRosa [fr...@studyblue.com](mailto:fr...@studyblue.com) wrote: > Hi, > > I've been doing a large number of updates and queries to my index over > the past couple of days. It currently contains about 10 million > documents which is about half the data I intend to insert. > > Twice today, my job locked up waiting on the result of a query. I had > to kill the java process and restart it. I have since started using > actionGet with the timeout parameter, but I'm worried as to what the > underlying cause of the lockup might be. It only happened twice over > many millions of operations. > > I obtained this stack trace the last time it happened, not sure if has > any helpful information: > > "Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on > condition [0x000000004446c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for \<0x00000000982e6728\> (a > org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java: > 811) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java: > 969) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java: > 1281) > at org.elasticsearch.common.util.concurrent.AbstractFuture > $Sync.get(AbstractFuture.java:249) > at > org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java: > 78) > at > org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java: > 44) > at com.studyblue.managers.impl.IdeaGraphManagerImpl > $GetIdeaThread.run(IdeaGraphManagerImpl.java:451) > > I have the stack traces from the rest of the threads if that would > help. Thanks. > > Frank ---
**Author:** ![Frank\_LaRosa](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@Frank\_LaRosa](https://discuss.elastic.co/u/Frank_LaRosa)\ **Post date:** [February 12, 2012, 4:05am UTC](https://discuss.elastic.co/t/lock-up-on-actionget/6678/3 "2012-02-12T04:05:17Z")
My server is reporting these errors: [2012-02-11 22:06:50,188][WARN][transport.netty] [es4.dev.studyblue.ec2] Exception caught on netty layer [[id: 0x4b01ea1e, /10.116.183.197:47872 =\> /10.80.83.88:9300]] java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:657) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java: 943) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java: 1336) at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java: 307) at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java: 207) at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java: 147) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java: 101) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java: 80) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java: 564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: 783) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java: 81) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java: 564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java: 559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java: 274) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java: 261) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java: 351) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java: 282) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: 202) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: 108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker $1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1110) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) [2012-02-11 22:07:15,399][WARN][index.engine.robin] [es4.dev.studyblue.ec2] [ideas][1] failed engine java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:657) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java: 943) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java: 1336) at org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBloomCache.java: 139) at org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java: 1266) at org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java: 392) at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java: 353) at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java: 295) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java: 142) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: 487) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction $1.run(TransportShardReplicationOperationAction.java:400) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1110) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) On Feb 11, 6:38 pm, Frank LaRosa [fr...@studyblue.com](mailto:fr...@studyblue.com) wrote: > Hi, > > I've been doing a large number of updates and queries to my index over > the past couple of days. It currently contains about 10 million > documents which is about half the data I intend to insert. > > Twice today, my job locked up waiting on the result of a query. I had > to kill the java process and restart it. I have since started using > actionGet with the timeout parameter, but I'm worried as to what the > underlying cause of the lockup might be. It only happened twice over > many millions of operations. > > I obtained this stack trace the last time it happened, not sure if has > any helpful information: > > "Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on > condition [0x000000004446c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for \<0x00000000982e6728\> (a > org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java: > 811) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java: > 969) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java: > 1281) > at org.elasticsearch.common.util.concurrent.AbstractFuture > $Sync.get(AbstractFuture.java:249) > at > org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java: > 78) > at > org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java: > 44) > at com.studyblue.managers.impl.IdeaGraphManagerImpl > $GetIdeaThread.run(IdeaGraphManagerImpl.java:451) > > I have the stack traces from the rest of the threads if that would > help. Thanks. > > Frank ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 12, 2012, 4:49pm UTC](https://discuss.elastic.co/t/lock-up-on-actionget/6678/4 "2012-02-12T16:49:45Z")
It seems like the nodes are running out of threads being created. How many concurrent search/read/index requests do you have? It might make sense to bound them using specific thread pool configuration: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/threadpool.html), On Sunday, February 12, 2012 at 6:05 AM, Frank LaRosa wrote: > My server is reporting these errors: > > [2012-02-11 22:06:50,188][WARN][transport.netty] > [es4.dev.studyblue.ec2] Exception caught on netty layer [[id: > 0x4b01ea1e, /10.116.183.197:47872 =\> /10.80.83.88:9300]] > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:657) > at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java: > 943) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java: > 1336) > at > org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java: > 307) > at > org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java: > 207) > at > org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java: > 147) > at > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java: > 101) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java: > 80) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java: > 564) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: > 783) > at > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java: > 81) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java: > 564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java: > 559) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java: > 274) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java: > 261) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java: > 351) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java: > 282) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java: > 202) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1110) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > [2012-02-11 22:07:15,399][WARN][index.engine.robin] > [es4.dev.studyblue.ec2] [ideas][1] failed engine > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:657) > at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java: > 943) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java: > 1336) > at > org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBloomCache.java: > 139) > at > org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java: > 1266) > at > org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java: > 392) > at > org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java: > 353) > at > org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java: > 295) > at > org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java: > 142) > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: > 487) > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction > $1.run(TransportShardReplicationOperationAction.java:400) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1110) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > > On Feb 11, 6:38 pm, Frank LaRosa \<[fr...@studyblue.com](mailto:fr...@studyblue.com) ([http://studyblue.com](http://studyblue.com))\> wrote: > > > Hi, > > > > I've been doing a large number of updates and queries to my index over > > the past couple of days. It currently contains about 10 million > > documents which is about half the data I intend to insert. > > > > Twice today, my job locked up waiting on the result of a query. I had > > to kill the java process and restart it. I have since started using > > actionGet with the timeout parameter, but I'm worried as to what the > > underlying cause of the lockup might be. It only happened twice over > > many millions of operations. > > > > I obtained this stack trace the last time it happened, not sure if has > > any helpful information: > > > > "Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on > > condition [0x000000004446c000] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for \<0x00000000982e6728\> (a > > org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java: > > 811) > > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java: > > 969) > > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java: > > 1281) > > at org.elasticsearch.common.util.concurrent.AbstractFuture > > $Sync.get(AbstractFuture.java:249) > > at > > org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java: > > 78) > > at > > org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java: > > 44) > > at com.studyblue.managers.impl.IdeaGraphManagerImpl > > $GetIdeaThread.run(IdeaGraphManagerImpl.java:451) > > > > I have the stack traces from the rest of the threads if that would > > help. Thanks. > > > > Frank ---
**Author:** ![lukeforehand\_2](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@lukeforehand\_2](https://discuss.elastic.co/u/lukeforehand_2)\ **Post date:** [February 12, 2012, 10:40pm UTC](https://discuss.elastic.co/t/lock-up-on-actionget/6678/5 "2012-02-12T22:40:09Z")
If you were doing millions of operations then it's possible that the unbounded search cache queued up past the number of threads your jvm could create. You can cap the size of the queues but your application has to handle the fallout when the queue becomes blocked. Here's the link to the different thread pool settings: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. -Luke On Feb 11, 10:05 pm, Frank LaRosa [fr...@studyblue.com](mailto:fr...@studyblue.com) wrote: > My server is reporting these errors: > > [2012-02-11 22:06:50,188][WARN][transport.netty] > [es4.dev.studyblue.ec2] Exception caught on netty layer [[id: > 0x4b01ea1e, /10.116.183.197:47872 =\> /10.80.83.88:9300]] > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:657) > at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java: > 943) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java: > 1336) > at > org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(Messa geChannelHandler.java: > 307) > at > org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChan nelHandler.java: > 207) > at > org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageC hannelHandler.java: > 147) > at > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(Mes sageChannelHandler.java: > 101) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleU pstream(SimpleChannelUpstreamHandler.java: > 80) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java: > 564) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: > 783) > at > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChann elsHandler.java: > 81) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java: > 564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java: > 559) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java: > 274) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java: > 261) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker. java: > 351) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelected Keys(NioWorker.java: > 282) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava: > 202) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java: > 108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > $1.run(DeadLockProofWorker.java:44) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1110) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > [2012-02-11 22:07:15,399][WARN][index.engine.robin] > [es4.dev.studyblue.ec2] [ideas][1] failed engine > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:657) > at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java: > 943) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java: > 1336) > at > org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBl oomCache.java: > 139) > at > org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromInde x(RobinEngine.java: > 1266) > at > org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.ja va: > 392) > at > org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java: > 353) > at > org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalInd exShard.java: > 295) > at > org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrim ary(TransportShardBulkAction.java: > 142) > at > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat ionAction.java: > 487) > at > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > $AsyncShardOperationAction > $1.run(TransportShardReplicationOperationAction.java:400) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1110) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > > On Feb 11, 6:38 pm, Frank LaRosa [fr...@studyblue.com](mailto:fr...@studyblue.com) wrote: > > > Hi, > > > I've been doing a large number of updates and queries to my index over > > the past couple of days. It currently contains about 10 million > > documents which is about half the data I intend to insert. > > > Twice today, my job locked up waiting on the result of a query. I had > > to kill the java process and restart it. I have since started using > > actionGetwith the timeout parameter, but I'm worried as to what the > > underlying cause of the lockup might be. It only happened twice over > > many millions of operations. > > > I obtained this stack trace the last time it happened, not sure if has > > any helpful information: > > > "Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on > > condition [0x000000004446c000] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for \<0x00000000982e6728\> (a > > org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java: > > 811) > > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java: > > 969) > > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java: > > 1281) > > at org.elasticsearch.common.util.concurrent.AbstractFuture > > $Sync.get(AbstractFuture.java:249) > > at > > org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java: > > 78) > > at > > org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java: > > 44) > > at com.studyblue.managers.impl.IdeaGraphManagerImpl > > $GetIdeaThread.run(IdeaGraphManagerImpl.java:451) > > > I have the stack traces from the rest of the threads if that would > > help. Thanks. > > > Frank ---
**Author:** ![Frank\_LaRosa](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@Frank\_LaRosa](https://discuss.elastic.co/u/Frank_LaRosa)\ **Post date:** [February 13, 2012, 2:33pm UTC](https://discuss.elastic.co/t/lock-up-on-actionget/6678/6 "2012-02-13T14:33:56Z")
My application was creating a lot of threads. The actual number varied according to the data, but in some cases, I'm sure I had several hundred. I've since added some code to limit the number of active threads to 16. That, in conjunction with allocating more memory on the servers, seems to have taken care of the problem. Thanks for your input. Frank On Feb 12, 10:49 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > It seems like the nodes are running out of threads being created. How many concurrent search/read/index requests do you have? It might make sense to bound them using specific thread pool configuration:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/threadpool.html), > > On Sunday, February 12, 2012 at 6:05 AM, Frank LaRosa wrote: > > > My server is reporting these errors: > > > [2012-02-11 22:06:50,188][WARN][transport.netty] > > [es4.dev.studyblue.ec2] Exception caught on netty layer [[id: > > 0x4b01ea1e, /10.116.183.197:47872 =\> /10.80.83.88:9300]] > > java.lang.OutOfMemoryError: unable to create new native thread > > at java.lang.Thread.start0(Native Method) > > at java.lang.Thread.start(Thread.java:657) > > at > > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java: > > 943) > > at > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java: > > 1336) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(Messa geChannelHandler.java: > > 307) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChan nelHandler.java: > > 207) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageC hannelHandler.java: > > 147) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(Mes sageChannelHandler.java: > > 101) > > at > > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleU pstream(SimpleChannelUpstreamHandler.java: > > 80) > > at > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java: > > 564) > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: > > 783) > > at > > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChann elsHandler.java: > > 81) > > at > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java: > > 564) > > at > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java: > > 559) > > at > > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java: > > 274) > > at > > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java: > > 261) > > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker. java: > > 351) > > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelected Keys(NioWorker.java: > > 282) > > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava: > > 202) > > at > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java: > > 108) > > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker > > $1.run(DeadLockProofWorker.java:44) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > > 1110) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:679) > > [2012-02-11 22:07:15,399][WARN][index.engine.robin] > > [es4.dev.studyblue.ec2] [ideas][1] failed engine > > java.lang.OutOfMemoryError: unable to create new native thread > > at java.lang.Thread.start0(Native Method) > > at java.lang.Thread.start(Thread.java:657) > > at > > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java: > > 943) > > at > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java: > > 1336) > > at > > org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBl oomCache.java: > > 139) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromInde x(RobinEngine.java: > > 1266) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.ja va: > > 392) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java: > > 353) > > at > > org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalInd exShard.java: > > 295) > > at > > org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrim ary(TransportShardBulkAction.java: > > 142) > > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat ionAction.java: > > 487) > > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > $AsyncShardOperationAction > > $1.run(TransportShardReplicationOperationAction.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > > 1110) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:679) > > > On Feb 11, 6:38 pm, Frank LaRosa \<[fr...@studyblue.com](mailto:fr...@studyblue.com) ([http://studyblue.com](http://studyblue.com))\> wrote: > > > > > Hi, > > > > I've been doing a large number of updates and queries to my index over > > > the past couple of days. It currently contains about 10 million > > > documents which is about half the data I intend to insert. > > > > Twice today, my job locked up waiting on the result of a query. I had > > > to kill the java process and restart it. I have since started using > > > actionGet with the timeout parameter, but I'm worried as to what the > > > underlying cause of the lockup might be. It only happened twice over > > > many millions of operations. > > > > I obtained this stack trace the last time it happened, not sure if has > > > any helpful information: > > > > "Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on > > > condition [0x000000004446c000] > > > java.lang.Thread.State: WAITING (parking) > > > at sun.misc.Unsafe.park(Native Method) > > > - parking to wait for \<0x00000000982e6728\> (a > > > org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java: > > > 811) > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java: > > > 969) > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java: > > > 1281) > > > at org.elasticsearch.common.util.concurrent.AbstractFuture > > > $Sync.get(AbstractFuture.java:249) > > > at > > > org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java: > > > 78) > > > at > > > org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java: > > > 44) > > > at com.studyblue.managers.impl.IdeaGraphManagerImpl > > > $GetIdeaThread.run(IdeaGraphManagerImpl.java:451) > > > > I have the stack traces from the rest of the threads if that would > > > help. Thanks. > > > > Frank ---
**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\ **Post date:** [February 13, 2012, 2:41pm UTC](https://discuss.elastic.co/t/lock-up-on-actionget/6678/7 "2012-02-13T14:41:23Z")
Also check your OS limits. For example on Linux: ulimit -u shows max number of processes per user (including threads) Default value is 1024 so you must usually extend this value before running ES node. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:39am UTC](https://discuss.elastic.co/t/lock-up-on-actionget/6678/8 "2017-07-06T03:39:37Z")