I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.
Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.
I obtained this stack trace the last time it happened, not sure if has
any helpful information:
"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)
I have the stack traces from the rest of the threads if that would
help. Thanks.
I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.
Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.
I obtained this stack trace the last time it happened, not sure if has
any helpful information:
"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)
I have the stack traces from the rest of the threads if that would
help. Thanks.
[2012-02-11 22:06:50,188][WARN ][transport.netty ]
[es4.dev.studyblue.ec2] Exception caught on netty layer [[id:
0x4b01ea1e, /10.116.183.197:47872 => /10.80.83.88:9300]]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:
307)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:
207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:
147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:
101)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-02-11 22:07:15,399][WARN ][index.engine.robin ]
[es4.dev.studyblue.ec2] [ideas][1] failed engine
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBloomCache.java:
139)
at
org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java:
1266)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:
392)
at
org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:
353)
at
org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:
295)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:
142)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:
487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction
$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.
Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.
I obtained this stack trace the last time it happened, not sure if has
any helpful information:
"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)
I have the stack traces from the rest of the threads if that would
help. Thanks.
It seems like the nodes are running out of threads being created. How many concurrent search/read/index requests do you have? It might make sense to bound them using specific thread pool configuration: Elasticsearch Platform — Find real-time answers at scale | Elastic,
On Sunday, February 12, 2012 at 6:05 AM, Frank LaRosa wrote:
My server is reporting these errors:
[2012-02-11 22:06:50,188][WARN ][transport.netty ]
[es4.dev.studyblue.ec2] Exception caught on netty layer [[id:
0x4b01ea1e, /10.116.183.197:47872 => /10.80.83.88:9300]]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:
307)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:
207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:
147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:
101)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-02-11 22:07:15,399][WARN ][index.engine.robin ]
[es4.dev.studyblue.ec2] [ideas][1] failed engine
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBloomCache.java:
139)
at
org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java:
1266)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:
392)
at
org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:
353)
at
org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:
295)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:
142)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:
487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction
$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.
Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.
I obtained this stack trace the last time it happened, not sure if has
any helpful information:
"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)
I have the stack traces from the rest of the threads if that would
help. Thanks.
If you were doing millions of operations then it's possible that the
unbounded search cache queued up past the number of threads your jvm
could create. You can cap the size of the queues but your application
has to handle the fallout when the queue becomes blocked.
Here's the link to the different thread pool settings:
[2012-02-11 22:06:50,188][WARN ][transport.netty ]
[es4.dev.studyblue.ec2] Exception caught on netty layer [[id:
0x4b01ea1e, /10.116.183.197:47872 => /10.80.83.88:9300]]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(Messa geChannelHandler.java:
307)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChan nelHandler.java:
207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageC hannelHandler.java:
147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(Mes sageChannelHandler.java:
101)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleU pstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChann elsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker. java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelected Keys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-02-11 22:07:15,399][WARN ][index.engine.robin ]
[es4.dev.studyblue.ec2] [ideas][1] failed engine
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBl oomCache.java:
139)
at
org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromInde x(RobinEngine.java:
1266)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.ja va:
392)
at
org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:
353)
at
org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalInd exShard.java:
295)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrim ary(TransportShardBulkAction.java:
142)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction
$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat ionAction.java:
487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction
$AsyncShardOperationAction
$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.
Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGetwith the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.
I obtained this stack trace the last time it happened, not sure if has
any helpful information:
"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)
I have the stack traces from the rest of the threads if that would
help. Thanks.
It seems like the nodes are running out of threads being created. How many concurrent search/read/index requests do you have? It might make sense to bound them using specific thread pool configuration:Elasticsearch Platform — Find real-time answers at scale | Elastic,
On Sunday, February 12, 2012 at 6:05 AM, Frank LaRosa wrote:
My server is reporting these errors:
[2012-02-11 22:06:50,188][WARN ][transport.netty ]
[es4.dev.studyblue.ec2] Exception caught on netty layer [[id:
0x4b01ea1e, /10.116.183.197:47872 => /10.80.83.88:9300]]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(Messa geChannelHandler.java:
307)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChan nelHandler.java:
207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageC hannelHandler.java:
147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(Mes sageChannelHandler.java:
101)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleU pstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChann elsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker. java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelected Keys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-02-11 22:07:15,399][WARN ][index.engine.robin ]
[es4.dev.studyblue.ec2] [ideas][1] failed engine
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1336)
at
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache.filter(SimpleBl oomCache.java:
139)
at
org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromInde x(RobinEngine.java:
1266)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.ja va:
392)
at
org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:
353)
at
org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalInd exShard.java:
295)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrim ary(TransportShardBulkAction.java:
142)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction
$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat ionAction.java:
487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction
$AsyncShardOperationAction
$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
I've been doing a large number of updates and queries to my index over
the past couple of days. It currently contains about 10 million
documents which is about half the data I intend to insert.
Twice today, my job locked up waiting on the result of a query. I had
to kill the java process and restart it. I have since started using
actionGet with the timeout parameter, but I'm worried as to what the
underlying cause of the lockup might be. It only happened twice over
many millions of operations.
I obtained this stack trace the last time it happened, not sure if has
any helpful information:
"Thread-2483135" prio=10 tid=0x00002aaabc192000 nid=0x4bf3 waiting on
condition [0x000000004446c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000982e6728> (a
org.elasticsearch.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:
811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterr uptibly(AbstractQueuedSynchronizer.java:
969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterrup tibly(AbstractQueuedSynchronizer.java:
1281)
at org.elasticsearch.common.util.concurrent.AbstractFuture
$Sync.get(AbstractFuture.java:249)
at
org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture. java:
78)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
44)
at com.studyblue.managers.impl.IdeaGraphManagerImpl
$GetIdeaThread.run(IdeaGraphManagerImpl.java:451)
I have the stack traces from the rest of the threads if that would
help. Thanks.
Also check your OS limits. For example on Linux:
ulimit -u
shows max number of processes per user (including threads)
Default value is 1024 so you must usually extend this value before running
ES node.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.