Lock up on actionGet

Frank_LaRosa · February 12, 2012, 12:38am

Hi,

I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.

Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.

I obtained this stack trace the last time it happened, not sure if has
any helpful information:

"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)

I have the stack traces from the rest of the threads if that would
help. Thanks.

Frank

Frank_LaRosa · February 12, 2012, 4:01am

After adding a timeout, I'm now seeing many hundreds of
"ElasticSearchTimeoutException".

I've stopped and restarted my application, but the error persists.

Any idea what's going on?

On Feb 11, 6:38 pm, Frank LaRosa fr...@studyblue.com wrote:

Hi,

I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.

Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.

I obtained this stack trace the last time it happened, not sure if has
any helpful information:

"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)

I have the stack traces from the rest of the threads if that would
help. Thanks.

Frank

Frank_LaRosa · February 12, 2012, 4:05am

My server is reporting these errors:

[2012-02-11 22:06:50,188][WARN ][transport.netty ]
[es4.dev.studyblue.ec2] Exception caught on netty layer [[id:
0x4b01ea1e, /10.116.183.197:47872 => /10.80.83.88:9300]]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:
307)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:
207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:
147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:
101)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-02-11 22:07:15,399][WARN ][index.engine.robin ]
[es4.dev.studyblue.ec2] [ideas][1] failed engine
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBloomCache.java:
139)
at
org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java:
1266)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:
392)
at
org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:
353)
at
org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:
295)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:
142)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:
487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction
$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

On Feb 11, 6:38 pm, Frank LaRosa fr...@studyblue.com wrote:

Hi,

I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.

Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.

I obtained this stack trace the last time it happened, not sure if has
any helpful information:

"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)

I have the stack traces from the rest of the threads if that would
help. Thanks.

Frank

kimchy · February 12, 2012, 4:49pm

It seems like the nodes are running out of threads being created. How many concurrent search/read/index requests do you have? It might make sense to bound them using specific thread pool configuration: Elasticsearch Platform — Find real-time answers at scale | Elastic,

On Sunday, February 12, 2012 at 6:05 AM, Frank LaRosa wrote:

My server is reporting these errors:

[2012-02-11 22:06:50,188][WARN ][transport.netty ]
[es4.dev.studyblue.ec2] Exception caught on netty layer [[id:
0x4b01ea1e, /10.116.183.197:47872 => /10.80.83.88:9300]]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:
307)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:
207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:
147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:
101)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-02-11 22:07:15,399][WARN ][index.engine.robin ]
[es4.dev.studyblue.ec2] [ideas][1] failed engine
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBloomCache.java:
139)
at
org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java:
1266)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:
392)
at
org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:
353)
at
org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:
295)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:
142)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:
487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction
$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

On Feb 11, 6:38 pm, Frank LaRosa <fr...@studyblue.com (http://studyblue.com)> wrote:

Hi,

I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.

Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.

I obtained this stack trace the last time it happened, not sure if has
any helpful information:

"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)

I have the stack traces from the rest of the threads if that would
help. Thanks.

Frank

lukeforehand_2 · February 12, 2012, 10:40pm

If you were doing millions of operations then it's possible that the
unbounded search cache queued up past the number of threads your jvm
could create. You can cap the size of the queues but your application
has to handle the fallout when the queue becomes blocked.

Here's the link to the different thread pool settings:

-Luke

On Feb 11, 10:05 pm, Frank LaRosa fr...@studyblue.com wrote:

My server is reporting these errors:

[2012-02-11 22:06:50,188][WARN ][transport.netty ]
[es4.dev.studyblue.ec2] Exception caught on netty layer [[id:
0x4b01ea1e, /10.116.183.197:47872 => /10.80.83.88:9300]]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(Messa geChannelHandler.java:
307)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChan nelHandler.java:
207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageC hannelHandler.java:
147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(Mes sageChannelHandler.java:
101)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleU pstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChann elsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker. java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelected Keys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-02-11 22:07:15,399][WARN ][index.engine.robin ]
[es4.dev.studyblue.ec2] [ideas][1] failed engine
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBl oomCache.java:
139)
at
org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromInde x(RobinEngine.java:
1266)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.ja va:
392)
at
org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:
353)
at
org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalInd exShard.java:
295)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrim ary(TransportShardBulkAction.java:
142)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction
$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat ionAction.java:
487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction
$AsyncShardOperationAction
$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

On Feb 11, 6:38 pm, Frank LaRosa fr...@studyblue.com wrote:

Hi,

I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.

Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGetwith the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.

I obtained this stack trace the last time it happened, not sure if has
any helpful information:

"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)

I have the stack traces from the rest of the threads if that would
help. Thanks.

Frank

Frank_LaRosa · February 13, 2012, 2:33pm

My application was creating a lot of threads. The actual number varied
according to the data, but in some cases, I'm sure I had several
hundred.

I've since added some code to limit the number of active threads to
16.

That, in conjunction with allocating more memory on the servers, seems
to have taken care of the problem.

Thanks for your input.

Frank

On Feb 12, 10:49 am, Shay Banon kim...@gmail.com wrote:

It seems like the nodes are running out of threads being created. How many concurrent search/read/index requests do you have? It might make sense to bound them using specific thread pool configuration:Elasticsearch Platform — Find real-time answers at scale | Elastic,

On Sunday, February 12, 2012 at 6:05 AM, Frank LaRosa wrote:

My server is reporting these errors:

[2012-02-11 22:06:50,188][WARN ][transport.netty ]
[es4.dev.studyblue.ec2] Exception caught on netty layer [[id:
0x4b01ea1e, /10.116.183.197:47872 => /10.80.83.88:9300]]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(Messa geChannelHandler.java:
307)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChan nelHandler.java:
207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageC hannelHandler.java:
147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(Mes sageChannelHandler.java:
101)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleU pstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChann elsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker. java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelected Keys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-02-11 22:07:15,399][WARN ][index.engine.robin ]
[es4.dev.studyblue.ec2] [ideas][1] failed engine
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBl oomCache.java:
139)
at
org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromInde x(RobinEngine.java:
1266)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.ja va:
392)
at
org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:
353)
at
org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalInd exShard.java:
295)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrim ary(TransportShardBulkAction.java:
142)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction
$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat ionAction.java:
487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction
$AsyncShardOperationAction
$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

On Feb 11, 6:38 pm, Frank LaRosa <fr...@studyblue.com (http://studyblue.com)> wrote:

Hi,

I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.

Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.

I obtained this stack trace the last time it happened, not sure if has
any helpful information:

"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)

I have the stack traces from the rest of the threads if that would
help. Thanks.

Frank

Wojciech_Durczynski · February 13, 2012, 2:41pm

Also check your OS limits. For example on Linux:
ulimit -u
shows max number of processes per user (including threads)
Default value is 1024 so you must usually extend this value before running
ES node.