My colleague is running into a strange memory error with elastic search on
his mac book. That seems to come and go. We've been seeing this error
pretty much the moment after I migrated to using the lucene 4.x snapshot
builds two months ago.
He's running pretty much an identical setup to what I have on a slightly
older macbook 13" (I have the 15"). Both have 8GB. I have an SSD, he has a
normal laptop drive. Both use the same jvm settings. Both have a very
recent sun jdk 1.7. Both have OSX 10.8.x (I've update to 10.8.3 last week,
he's still on 10.8.2).
He's seeing these errors randomly. I've never seen them. Unlike him, I
actually use the index (see my other post yesterday ;-)). All he does is
run a maven build that launches es, inserts a hand ful of test objects and
then shuts down es. We start es with an in memory index in our tests.
We both use these settings:
export
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_13.jdk/Contents/Home
PATH=$JAVA_HOME/bin:/usr/local/mvn/bin:$JRUBY_HOME/bin:$PATH; export PATH
MAVEN_OPTS='-Xmx2048m -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled
-Djava.awt.headless=true'
We've been seeing this error on and of on his machine for a few months now
and I've been struggling to figure out what the problem is since I can't
reproduce on my own machine. 2 GB is more than enough for running our
tests. When I run the same build it peaks out at a mere 225MB in jvisualvm.
The direct buffer memory appears to be some shared memory buffer.
So, has anyone seen similar errors?
Here's the stacktrace:
INFO [elasticsearch[devnode][clusterService#updateTask][T#1]]
(Log4jESLogger.java:104) - [devnode] [localstream] creating index, cause
[auto(index api)], shards [5]/[1], mappings []
INFO [elasticsearch[devnode][generic][T#2]] (Log4jESLogger.java:104) -
[devnode] [localstream][0] deleting shard content
INFO [elasticsearch[devnode][generic][T#2]] (Log4jESLogger.java:104) -
[devnode] [localstream][1] deleting shard content
INFO [elasticsearch[devnode][generic][T#2]] (Log4jESLogger.java:104) -
[devnode] [localstream][2] deleting shard content
INFO [elasticsearch[devnode][generic][T#2]] (Log4jESLogger.java:104) -
[devnode] [localstream][3] deleting shard content
INFO [elasticsearch[devnode][generic][T#2]] (Log4jESLogger.java:104) -
[devnode] [localstream][4] deleting shard content
INFO [elasticsearch[devnode][clusterService#updateTask][T#1]]
(Log4jESLogger.java:104) - [devnode] [localstream] update_mapping [poi]
(dynamic)
INFO [elasticsearch[devnode][clusterService#updateTask][T#1]]
(Log4jESLogger.java:104) - [devnode] [posts] update_mapping [post] (dynamic)
WARN [elasticsearch[devnode][refresh][T#3]] (Log4jESLogger.java:119) -
[devnode] [posts][0] failed engine
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at
org.apache.lucene.store.bytebuffer.PlainByteBufferAllocator.allocate(PlainByteBufferAllocator.java:55)
at
org.apache.lucene.store.bytebuffer.CachingByteBufferAllocator.allocate(CachingByteBufferAllocator.java:52)
at
org.elasticsearch.cache.memory.ByteBufferCache.allocate(ByteBufferCache.java:101)
at
org.apache.lucene.store.bytebuffer.ByteBufferIndexOutput.switchCurrentBuffer(ByteBufferIndexOutput.java:106)
at
org.apache.lucene.store.bytebuffer.ByteBufferIndexOutput.writeBytes(ByteBufferIndexOutput.java:93)
at
org.elasticsearch.common.lucene.store.BufferedChecksumIndexOutput.flushBuffer(BufferedChecksumIndexOutput.java:65)
at
org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)
at
org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:102)
at
org.elasticsearch.common.lucene.store.BufferedChecksumIndexOutput.flush(BufferedChecksumIndexOutput.java:76)
at
org.apache.lucene.store.BufferedIndexOutput.close(BufferedIndexOutput.java:126)
at
org.elasticsearch.common.lucene.store.BufferedChecksumIndexOutput.close(BufferedChecksumIndexOutput.java:59)
at
org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:545)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:146)
at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesConsumer.close(Lucene42DocValuesConsumer.java:162)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:146)
at org.apache.lucene.index.NormsConsumer.flush(NormsConsumer.java:67)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:54)
at
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81)
at
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:493)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:422)
at
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:559)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:357)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:270)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:245)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:235)
at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:169)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:118)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
at
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:155)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:204)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:763)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:402)
at
org.elasticsearch.action.admin.indices.refresh.TransportRefreshAction.shardOperation(TransportRefreshAction.java:120)
at
org.elasticsearch.action.admin.indices.refresh.TransportRefreshAction.shardOperation(TransportRefreshAction.java:49)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:265)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:242)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:218)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Suppressed: java.lang.OutOfMemoryError: Direct buffer memory
... 43 more
WARN [elasticsearch[devnode][generic][T#2]] (Log4jESLogger.java:114) -
[devnode] sending failed shard for [posts][0],
node[fhtPH4u0TmS9dhSjudUCJQ], [P], s[STARTED], reason [engine failure,
message [OutOfMemoryError[Direct buffer memory]]]
WARN [elasticsearch[devnode][generic][T#2]] (Log4jESLogger.java:114) -
[devnode] received shard failed for [posts][0],
node[fhtPH4u0TmS9dhSjudUCJQ], [P], s[STARTED], reason [engine failure,
message [OutOfMemoryError[Direct buffer memory]]]
Tests run: 142, Failures: 2, Errors: 0, Skipped: 74, Time elapsed: 70.052
sec <<< FAILURE!
INFO [Thread-1] (Log4jESLogger.java:104) - [devnode]
{0.90.0.RC2-SNAPSHOT}[966]: stopping ...
WARN [Thread-1] (Log4jESLogger.java:119) - An exception was thrown by an
exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been
shutdown
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:115)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:71)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:55)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:632)
at
org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at
org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:654)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:777)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:725)
at
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
at
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:152)
at
org.elasticsearch.rest.action.index.RestIndexAction$1.onFailure(RestIndexAction.java:139)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onClose(TransportShardReplicationOperationAction.java:497)
at
org.elasticsearch.cluster.service.InternalClusterService.doStop(InternalClusterService.java:128)
at
org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105)
at org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:250)
at org.elasticsearch.node.internal.InternalNode.close(InternalNode.java:269)
at com.localstream.es.EsLauncher$1.run(EsLauncher.java:31)
INFO [Thread-2] (AbstractApplicationContext.java:1042) - Closing
org.springframework.context.support.GenericApplicationContext@1e6c8a0c:
startup date [Thu Mar 21 11:04:16 CET 2013]; root of context hierarchy
INFO [Thread-1] (Log4jESLogger.java:104) - [devnode]
{0.90.0.RC2-SNAPSHOT}[966]: stopped
INFO [Thread-2] (DefaultSingletonBeanRegistry.java:444) - Destroying
singletons in
org.springframework.beans.factory.support.DefaultListableBeanFactory@632270ff:
defining beans
[org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,springConfig,embeddedESLauncherConfig,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor,com.localstream.server.spring.ConfigurationConfig,configuration,esRestClient,parser,placesDAO,esPostDAO,esUserDAO,httpClient];
root of factory hierarchy
INFO [Thread-1] (Log4jESLogger.java:104) - [devnode]
{0.90.0.RC2-SNAPSHOT}[966]: closing ...
INFO [Thread-1] (Log4jESLogger.java:104) - [devnode]
{0.90.0.RC2-SNAPSHOT}[966]: closed
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.