# Out of Memory - if index.store.fs.memory.enabled is true **URL:** \ **Category:** Elasticsearch\ **Created:** [June 25, 2010, 11:00pm UTC](https://discuss.elastic.co/t/out-of-memory-if-index-store-fs-memory-enabled-is-true/3043 "2010-06-25T23:00:11Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Diptamay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diptamay/32/2805_2.png) [@Diptamay](https://discuss.elastic.co/u/Diptamay)\ **Post date:** [June 25, 2010, 11:00pm UTC](https://discuss.elastic.co/t/out-of-memory-if-index-store-fs-memory-enabled-is-true/3043/1 "2010-06-25T23:00:11Z")
Just to see what are the limits, I started 2 nodes (each with default heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a pre-created index of like 50k documents, with the following configuration (in specific I had set index.store.fs.memory.enabled=true for this run) gateway: type: fs fs: location: /Users/sanyal/Documents/workspace/hb\_indices index: gateway: snapshot\_interval : 30s store: type: niofs fs: memory: enabled: true number\_of\_shards : 3 number\_of\_replicas : 2 path: logs: /Users/sanyal/Documents/workspace/logs By enabling the direct memory buffer, I see out of memory errors on the master node. I don't get the errors if I don't set index.store.fs.memory.enabled=true. Am I using it incorrectly, or is this something which should not be used ideally? * * * [17:51:45,685][INFO][node] [Power, James Dr.] {ElasticSearch/0.8.0}[86452]: Initializing ... [17:51:45,689][INFO][plugins] [Power, James Dr.] Loaded [] [17:51:47,011][INFO][node] [Power, James Dr.] {ElasticSearch/0.8.0}[86452]: Initialized [17:51:47,012][INFO][node] [Power, James Dr.] {ElasticSearch/0.8.0}[86452]: Starting ... [17:51:47,335][INFO][transport] [Power, James Dr.] bound\_address[inet[/0.0.0.0:9300]], publish\_address[inet[/ 169.254.180.96:9300]] [17:51:50,618][INFO][cluster.service] [Power, James Dr.] New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110] [inet[/169.254.180.96:9300]], Reason: zen-disco- initial\_connect(master) [17:51:50,783][INFO][discovery] [Power, James Dr.] elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110 [17:51:50,890][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_175], cause [gateway], shards [3]/[2], mappings [named-search, stylesheet, view, mm-iphone-alert] [17:51:52,455][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_168], cause [gateway], shards [3]/[2], mappings [video-topic-page, view, video-clip-list] [17:51:53,512][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_164], cause [gateway], shards [3]/[2], mappings [dam-view, video-topic-page, stylesheet, view, video-clip-list] [17:51:54,595][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_165], cause [gateway], shards [3]/[2], mappings [view] [17:51:55,589][INFO][cluster.service] [Power, James Dr.] Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ 169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ 169.254.180.96:9301]]]) [17:51:55,685][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_161], cause [gateway], shards [3]/[2], mappings [view] [17:51:56,872][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_62], cause [gateway], shards [3]/[2], mappings [dam- view, video-topic-page, named-search, stylesheet, view, video-clip- list] [17:52:02,708][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_63], cause [gateway], shards [3]/[2], mappings [view] [17:52:05,479][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_170], cause [gateway], shards [3]/[2], mappings [view] [17:52:07,431][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_14], cause [gateway], shards [3]/[2], mappings [pressbox-asset-list, pressbox-asset, dam-view, audio, named-search, lineup-player, section, vpp-nav-list, article, vpp-promowall, page, video-topic-page, stylesheet, view, video-clip-list, note, lineup, video] [17:52:10,657][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_8], cause [gateway], shards [3]/[2], mappings [video-topic-page, view, video-clip-list] [17:52:13,410][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_9], cause [gateway], shards [3]/[2], mappings [view] [17:52:16,668][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_33], cause [gateway], shards [3]/[2], mappings [video-topic-page, view, video-clip-list] [17:52:20,923][WARN][gateway] [Power, James Dr.] Waited for 30s for indices to be created from the gateway, and not all have been created [17:52:22,477][INFO][http] [Power, James Dr.] bound\_address[inet[/0.0.0.0:9201]], publish\_address[inet[/ 169.254.180.96:9201]] [17:52:23,638][INFO][jmx] [Power, James Dr.] bound\_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi], publish\_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/ jmxrmi] [17:52:24,021][INFO][node] [Power, James Dr.] {ElasticSearch/0.8.0}[86452]: Started [17:52:24,576][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_34], cause [gateway], shards [3]/[2], mappings [video-topic-page, view, video-clip-list] [17:52:30,201][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_43], cause [gateway], shards [3]/[2], mappings [view] [17:52:33,944][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_179], cause [gateway], shards [3]/[2], mappings [named-search, view] [17:52:42,696][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_44], cause [gateway], shards [3]/[2], mappings [view] [17:53:05,119][INFO][cluster.metadata] [Power, James Dr.] Creating Index [hb\_41], cause [gateway], shards [3]/[2], mappings [view] [17:53:54,451][WARN][indices.cluster] [Power, James Dr.] Failed to create shard for index [hb\_41] and shard id [2] org.elasticsearch.util.inject.CreationException: Guice creation errors: 1. Error injecting constructor, java.lang.OutOfMemoryError: Direct buffer memory at org.elasticsearch.index.store.fs.NioFsStore.(Unknown Source) while locating org.elasticsearch.index.store.fs.NioFsStore while locating org.elasticsearch.index.store.Store for parameter 2 at org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown Source) while locating org.elasticsearch.index.shard.service.InternalIndexShard while locating org.elasticsearch.index.shard.service.IndexShard Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:633) at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) at org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteBufferDirectory.java: 2. ``` at ``` org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferDirectory.java: 110) at org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeeded(AbstractFsStore.java: 70) at org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java: 63) at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java: 27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.elasticsearch.util.inject.DefaultConstructionProxyFactory $1.newInstance(DefaultConstructionProxyFactory.java:53) at org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInjector.java: 86) at org.elasticsearch.util.inject.ConstructorBindingImpl $Factory.get(ConstructorBindingImpl.java:97) at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java: 56) at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter $1.call(ProviderToInternalFactoryAdapter.java:45) at org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: 791) at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java: 42) at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54) at org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java: 48) at org.elasticsearch.util.inject.SingleParameterInjector.inject(SingleParameterInjector.java: 42) at org.elasticsearch.util.inject.SingleParameterInjector.getAll(SingleParameterInjector.java: 66) at org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInjector.java: 85) at org.elasticsearch.util.inject.ConstructorBindingImpl $Factory.get(ConstructorBindingImpl.java:97) at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java: 56) at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter $1.call(ProviderToInternalFactoryAdapter.java:45) at org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: 791) at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java: 42) at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54) at org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java: 48) at org.elasticsearch.util.inject.InjectorBuilder $1.call(InjectorBuilder.java:197) at org.elasticsearch.util.inject.InjectorBuilder $1.call(InjectorBuilder.java:191) at org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: 784) at org.elasticsearch.util.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java: 191) at org.elasticsearch.util.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java: 173) at org.elasticsearch.util.inject.InjectorBuilder.build(InjectorBuilder.java: 110) at org.elasticsearch.util.inject.InjectorImpl.createChildInjector(InjectorImpl.java: 124) at org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java: 224) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java: 286) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewShards(IndicesClusterStateService.java: 255) at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java: 112) at org.elasticsearch.cluster.service.InternalClusterService $2.run(InternalClusterService.java:198) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:637) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 26, 2010, 7:36pm UTC](https://discuss.elastic.co/t/out-of-memory-if-index-store-fs-memory-enabled-is-true/3043/2 "2010-06-26T19:36:20Z")
My guess is that you create several indices, and the memory store cache is done on a per shard level. I have changed it to be done on a node level, explained here: [Memory Store / FS Memory: Create a node level memory store cache and allocator · Issue #235 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/235). Stil, this is only an assumption, the other thing that I can think of is you using 32 bit JVM? -shay.banon On Sat, Jun 26, 2010 at 2:00 AM, diptamay [diptamay@gmail.com](mailto:diptamay@gmail.com) wrote: > Just to see what are the limits, I started 2 nodes (each with default > heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a > pre-created index of like 50k documents, with the following > configuration (in specific I had set > index.store.fs.memory.enabled=true for this run) > > gateway: > type: fs > fs: > location: /Users/sanyal/Documents/workspace/hb\_indices > index: > gateway: > snapshot\_interval : 30s > store: > type: niofs > fs: > memory: > enabled: true > number\_of\_shards : 3 > number\_of\_replicas : 2 > path: > logs: /Users/sanyal/Documents/workspace/logs > > By enabling the direct memory buffer, I see out of memory errors on > the master node. I don't get the errors if I don't set > index.store.fs.memory.enabled=true. > > Am I using it incorrectly, or is this something which should not be > used ideally? > > * * * > > [17:51:45,685][INFO][node] [Power, James Dr.] > {Elasticsearch/0.8.0}[86452]: Initializing ... > [17:51:45,689][INFO][plugins] [Power, James Dr.] > Loaded > [17:51:47,011][INFO][node] [Power, James Dr.] > {Elasticsearch/0.8.0}[86452]: Initialized > [17:51:47,012][INFO][node] [Power, James Dr.] > {Elasticsearch/0.8.0}[86452]: Starting ... > [17:51:47,335][INFO][transport] [Power, James Dr.] > bound\_address[inet[/0.0.0.0:9300]], publish\_address[inet[/ > 169.254.180.96:9300]] > [17:51:50,618][INFO][cluster.service] [Power, James Dr.] > New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110] > [inet[/169.254.180.96:9300]], Reason: zen-disco- > initial\_connect(master) > [17:51:50,783][INFO][discovery] [Power, James Dr.] > elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110 > [17:51:50,890][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_175], cause [gateway], shards [3]/[2], mappings > [named-search, stylesheet, view, mm-iphone-alert] > [17:51:52,455][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_168], cause [gateway], shards [3]/[2], mappings > [video-topic-page, view, video-clip-list] > [17:51:53,512][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_164], cause [gateway], shards [3]/[2], mappings > [dam-view, video-topic-page, stylesheet, view, video-clip-list] > [17:51:54,595][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_165], cause [gateway], shards [3]/[2], mappings > [view] > [17:51:55,589][INFO][cluster.service] [Power, James Dr.] > Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ > 169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human > Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ > 169.254.180.96:9301]]]) > [17:51:55,685][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_161], cause [gateway], shards [3]/[2], mappings > [view] > [17:51:56,872][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_62], cause [gateway], shards [3]/[2], mappings [dam- > view, video-topic-page, named-search, stylesheet, view, video-clip- > list] > [17:52:02,708][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_63], cause [gateway], shards [3]/[2], mappings > [view] > [17:52:05,479][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_170], cause [gateway], shards [3]/[2], mappings > [view] > [17:52:07,431][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_14], cause [gateway], shards [3]/[2], mappings > [pressbox-asset-list, pressbox-asset, dam-view, audio, named-search, > lineup-player, section, vpp-nav-list, article, vpp-promowall, page, > video-topic-page, stylesheet, view, video-clip-list, note, lineup, > video] > [17:52:10,657][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_8], cause [gateway], shards [3]/[2], mappings > [video-topic-page, view, video-clip-list] > [17:52:13,410][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_9], cause [gateway], shards [3]/[2], mappings > [view] > [17:52:16,668][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_33], cause [gateway], shards [3]/[2], mappings > [video-topic-page, view, video-clip-list] > [17:52:20,923][WARN][gateway] [Power, James Dr.] > Waited for 30s for indices to be created from the gateway, and not all > have been created > [17:52:22,477][INFO][http] [Power, James Dr.] > bound\_address[inet[/0.0.0.0:9201]], publish\_address[inet[/ > 169.254.180.96:9201]] > [17:52:23,638][INFO][jmx] [Power, James Dr.] > bound\_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi], > publish\_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/ > jmxrmi] > [17:52:24,021][INFO][node] [Power, James Dr.] > {Elasticsearch/0.8.0}[86452]: Started > [17:52:24,576][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_34], cause [gateway], shards [3]/[2], mappings > [video-topic-page, view, video-clip-list] > [17:52:30,201][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_43], cause [gateway], shards [3]/[2], mappings > [view] > [17:52:33,944][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_179], cause [gateway], shards [3]/[2], mappings > [named-search, view] > [17:52:42,696][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_44], cause [gateway], shards [3]/[2], mappings > [view] > [17:53:05,119][INFO][cluster.metadata] [Power, James Dr.] > Creating Index [hb\_41], cause [gateway], shards [3]/[2], mappings > [view] > [17:53:54,451][WARN][indices.cluster] [Power, James Dr.] > Failed to create shard for index [hb\_41] and shard id [2] > org.elasticsearch.util.inject.CreationException: Guice creation > errors: > > 1. Error injecting constructor, java.lang.OutOfMemoryError: Direct > buffer memory > at org.elasticsearch.index.store.fs.NioFsStore.(Unknown > Source) > while locating org.elasticsearch.index.store.fs.NioFsStore > while locating org.elasticsearch.index.store.Store > for parameter 2 at > org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown > Source) > while locating > org.elasticsearch.index.shard.service.InternalIndexShard > while locating org.elasticsearch.index.shard.service.IndexShard > Caused by: java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:633) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) > at > > org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteBufferDirectory.java: > 247) > at > > org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferDirectory.java: > 110) > at > > org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeeded(AbstractFsStore.java: > 70) > at > org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java: > 63) > at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown > Source) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java: > 27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at org.elasticsearch.util.inject.DefaultConstructionProxyFactory > $1.newInstance(DefaultConstructionProxyFactory.java:53) > at > > org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInjector.java: > 86) > at org.elasticsearch.util.inject.ConstructorBindingImpl > $Factory.get(ConstructorBindingImpl.java:97) > at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java: > 56) > at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter > $1.call(ProviderToInternalFactoryAdapter.java:45) > at > org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: > 791) > at > > org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java: > 42) > at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54) > at > > org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java: > 48) > at > > org.elasticsearch.util.inject.SingleParameterInjector.inject(SingleParameterInjector.java: > 42) > at > > org.elasticsearch.util.inject.SingleParameterInjector.getAll(SingleParameterInjector.java: > 66) > at > > org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInjector.java: > 85) > at org.elasticsearch.util.inject.ConstructorBindingImpl > $Factory.get(ConstructorBindingImpl.java:97) > at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java: > 56) > at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter > $1.call(ProviderToInternalFactoryAdapter.java:45) > at > org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: > 791) > at > > org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java: > 42) > at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54) > at > > org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java: > 48) > at org.elasticsearch.util.inject.InjectorBuilder > $1.call(InjectorBuilder.java:197) > at org.elasticsearch.util.inject.InjectorBuilder > $1.call(InjectorBuilder.java:191) > at > org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: > 784) > at > > org.elasticsearch.util.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java: > 191) > at > > org.elasticsearch.util.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java: > 173) > at > org.elasticsearch.util.inject.InjectorBuilder.build(InjectorBuilder.java: > 110) > at > > org.elasticsearch.util.inject.InjectorImpl.createChildInjector(InjectorImpl.java: > 124) > at > > org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java: > 224) > at > > org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java: > 286) > at > > org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewShards(IndicesClusterStateService.java: > 255) > at > > org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java: > 112) > at org.elasticsearch.cluster.service.InternalClusterService > $2.run(InternalClusterService.java:198) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:637) ---
**Author:** ![Diptamay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diptamay/32/2805_2.png) [@Diptamay](https://discuss.elastic.co/u/Diptamay)\ **Post date:** [June 26, 2010, 10:39pm UTC](https://discuss.elastic.co/t/out-of-memory-if-index-store-fs-memory-enabled-is-true/3043/3 "2010-06-26T22:39:58Z")
Hi Shay Yes I created several indices, around 20 or so, since our CMS app supports multiple sites and I wanted the indices to be per site. However, out of the 50k documents I indexed, majority like 46k or so belonged to only 1 index. Thanks for the explanation of the issue (I started both the nodes with 3 shards on them.). I could try out the fix by checking out the latest, and let you know? By the way, I am using the 64-bit jvm on my snow leopard (64-bit). Thanks Diptamay On Jun 26, 3:36 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > My guess is that you create several indices, and the memory store cache is > done on a per shard level. I have changed it to be done on a node level, > explained here:[Memory Store / FS Memory: Create a node level memory store cache and allocator · Issue #235 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/235). Stil, this > is only an assumption, the other thing that I can think of is you using 32 > bit JVM? > > -shay.banon > > On Sat, Jun 26, 2010 at 2:00 AM, diptamay [dipta...@gmail.com](mailto:dipta...@gmail.com) wrote: > > > Just to see what are the limits, I started 2 nodes (each with default > > heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a > > pre-created index of like 50k documents, with the following > > configuration (in specific I had set > > index.store.fs.memory.enabled=true for this run) > > > gateway: > > type: fs > > fs: > > location: /Users/sanyal/Documents/workspace/hb\_indices > > index: > > gateway: > > snapshot\_interval : 30s > > store: > > type: niofs > > fs: > > memory: > > enabled: true > > number\_of\_shards : 3 > > number\_of\_replicas : 2 > > path: > > logs: /Users/sanyal/Documents/workspace/logs > > > By enabling the direct memory buffer, I see out of memory errors on > > the master node. I don't get the errors if I don't set > > index.store.fs.memory.enabled=true. > > > Am I using it incorrectly, or is this something which should not be > > used ideally? > > > * * * > > > [17:51:45,685][INFO][node] [Power, James Dr.] > > {Elasticsearch/0.8.0}[86452]: Initializing ... > > [17:51:45,689][INFO][plugins] [Power, James Dr.] > > Loaded > > [17:51:47,011][INFO][node] [Power, James Dr.] > > {Elasticsearch/0.8.0}[86452]: Initialized > > [17:51:47,012][INFO][node] [Power, James Dr.] > > {Elasticsearch/0.8.0}[86452]: Starting ... > > [17:51:47,335][INFO][transport] [Power, James Dr.] > > bound\_address[inet[/0.0.0.0:9300]], publish\_address[inet[/ > > 169.254.180.96:9300]] > > [17:51:50,618][INFO][cluster.service] [Power, James Dr.] > > New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110] > > [inet[/169.254.180.96:9300]], Reason: zen-disco- > > initial\_connect(master) > > [17:51:50,783][INFO][discovery] [Power, James Dr.] > > elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110 > > [17:51:50,890][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_175], cause [gateway], shards [3]/[2], mappings > > [named-search, stylesheet, view, mm-iphone-alert] > > [17:51:52,455][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_168], cause [gateway], shards [3]/[2], mappings > > [video-topic-page, view, video-clip-list] > > [17:51:53,512][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_164], cause [gateway], shards [3]/[2], mappings > > [dam-view, video-topic-page, stylesheet, view, video-clip-list] > > [17:51:54,595][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_165], cause [gateway], shards [3]/[2], mappings > > [view] > > [17:51:55,589][INFO][cluster.service] [Power, James Dr.] > > Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ > > 169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human > > Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ > > 169.254.180.96:9301]]]) > > [17:51:55,685][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_161], cause [gateway], shards [3]/[2], mappings > > [view] > > [17:51:56,872][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_62], cause [gateway], shards [3]/[2], mappings [dam- > > view, video-topic-page, named-search, stylesheet, view, video-clip- > > list] > > [17:52:02,708][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_63], cause [gateway], shards [3]/[2], mappings > > [view] > > [17:52:05,479][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_170], cause [gateway], shards [3]/[2], mappings > > [view] > > [17:52:07,431][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_14], cause [gateway], shards [3]/[2], mappings > > [pressbox-asset-list, pressbox-asset, dam-view, audio, named-search, > > lineup-player, section, vpp-nav-list, article, vpp-promowall, page, > > video-topic-page, stylesheet, view, video-clip-list, note, lineup, > > video] > > [17:52:10,657][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_8], cause [gateway], shards [3]/[2], mappings > > [video-topic-page, view, video-clip-list] > > [17:52:13,410][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_9], cause [gateway], shards [3]/[2], mappings > > [view] > > [17:52:16,668][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_33], cause [gateway], shards [3]/[2], mappings > > [video-topic-page, view, video-clip-list] > > [17:52:20,923][WARN][gateway] [Power, James Dr.] > > Waited for 30s for indices to be created from the gateway, and not all > > have been created > > [17:52:22,477][INFO][http] [Power, James Dr.] > > bound\_address[inet[/0.0.0.0:9201]], publish\_address[inet[/ > > 169.254.180.96:9201]] > > [17:52:23,638][INFO][jmx] [Power, James Dr.] > > bound\_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi], > > publish\_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/ > > jmxrmi] > > [17:52:24,021][INFO][node] [Power, James Dr.] > > {Elasticsearch/0.8.0}[86452]: Started > > [17:52:24,576][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_34], cause [gateway], shards [3]/[2], mappings > > [video-topic-page, view, video-clip-list] > > [17:52:30,201][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_43], cause [gateway], shards [3]/[2], mappings > > [view] > > [17:52:33,944][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_179], cause [gateway], shards [3]/[2], mappings > > [named-search, view] > > [17:52:42,696][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_44], cause [gateway], shards [3]/[2], mappings > > [view] > > [17:53:05,119][INFO][cluster.metadata] [Power, James Dr.] > > Creating Index [hb\_41], cause [gateway], shards [3]/[2], mappings > > [view] > > [17:53:54,451][WARN][indices.cluster] [Power, James Dr.] > > Failed to create shard for index [hb\_41] and shard id [2] > > org.elasticsearch.util.inject.CreationException: Guice creation > > errors: > > > 1. Error injecting constructor, java.lang.OutOfMemoryError: Direct > > buffer memory > > at org.elasticsearch.index.store.fs.NioFsStore.(Unknown > > Source) > > while locating org.elasticsearch.index.store.fs.NioFsStore > > while locating org.elasticsearch.index.store.Store > > for parameter 2 at > > org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown > > Source) > > while locating > > org.elasticsearch.index.shard.service.InternalIndexShard > > while locating org.elasticsearch.index.shard.service.IndexShard > > Caused by: java.lang.OutOfMemoryError: Direct buffer memory > > at java.nio.Bits.reserveMemory(Bits.java:633) > > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95) > > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) > > at > > > org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteB ufferDirectory.java: > > 247) > > at > > > org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferD irectory.java: > > 110) > > at > > > org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeed ed(AbstractFsStore.java: > > 70) > > at > > org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java: > > 63) > > at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown > > Source) > > at > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru ctorAccessorImpl.java: > > 27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at org.elasticsearch.util.inject.DefaultConstructionProxyFactory > > $1.newInstance(DefaultConstructionProxyFactory.java:53) > > at > > > org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje ctor.java: > > 86) > > at org.elasticsearch.util.inject.ConstructorBindingImpl > > $Factory.get(ConstructorBindingImpl.java:97) > > at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java: > > 56) > > at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter > > $1.call(ProviderToInternalFactoryAdapter.java:45) > > at > > org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: > > 791) > > at > > > org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(Provider ToInternalFactoryAdapter.java: > > 42) > > at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54) > > at > > > org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(Internal FactoryToProviderAdapter.java: > > 48) > > at > > > org.elasticsearch.util.inject.SingleParameterInjector.inject(SingleParamete rInjector.java: > > 42) > > at > > > org.elasticsearch.util.inject.SingleParameterInjector.getAll(SingleParamete rInjector.java: > > 66) > > at > > > org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje ctor.java: > > 85) > > at org.elasticsearch.util.inject.ConstructorBindingImpl > > $Factory.get(ConstructorBindingImpl.java:97) > > at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java: > > 56) > > at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter > > $1.call(ProviderToInternalFactoryAdapter.java:45) > > at > > org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: > > 791) > > at > > > org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(Provider ToInternalFactoryAdapter.java: > > 42) > > at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54) > > at > > ... > > read more » ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 26, 2010, 11:01pm UTC](https://discuss.elastic.co/t/out-of-memory-if-index-store-fs-memory-enabled-is-true/3043/4 "2010-06-26T23:01:41Z")
The size of the index is not relevant in this case, since the "warm\_cache" is set to true, each shard will pre-populate a 20mb size of native buffer. So, in your case, 20mb \* 20 \* 2 (I think you used 2 shards per index) is 800mb used OOB. You can set the warm\_cache setting to disable it on a per shard level, or move to master, where 20mb will be pre allocated on the node level, and not on a per shard level. -shay.banon On Sun, Jun 27, 2010 at 1:39 AM, diptamay [diptamay@gmail.com](mailto:diptamay@gmail.com) wrote: > Hi Shay > > Yes I created several indices, around 20 or so, since our CMS app > supports multiple sites and I wanted the indices to be per site. > However, out of the 50k documents I indexed, majority like 46k or so > belonged to only 1 index. > > Thanks for the explanation of the issue (I started both the nodes with > 3 shards on them.). I could try out the fix by checking out the > latest, and let you know? > > By the way, I am using the 64-bit jvm on my snow leopard (64-bit). > > Thanks > Diptamay > > On Jun 26, 3:36 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > My guess is that you create several indices, and the memory store cache > > is > > done on a per shard level. I have changed it to be done on a node level, > > explained here: > > [Memory Store / FS Memory: Create a node level memory store cache and allocator · Issue #235 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/235). Stil, this > > is only an assumption, the other thing that I can think of is you using > > 32 > > bit JVM? > > > > -shay.banon > > > > On Sat, Jun 26, 2010 at 2:00 AM, diptamay [dipta...@gmail.com](mailto:dipta...@gmail.com) wrote: > > > > > Just to see what are the limits, I started 2 nodes (each with default > > > heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a > > > pre-created index of like 50k documents, with the following > > > configuration (in specific I had set > > > index.store.fs.memory.enabled=true for this run) > > > > > gateway: > > > type: fs > > > fs: > > > location: /Users/sanyal/Documents/workspace/hb\_indices > > > index: > > > gateway: > > > snapshot\_interval : 30s > > > store: > > > type: niofs > > > fs: > > > memory: > > > enabled: true > > > number\_of\_shards : 3 > > > number\_of\_replicas : 2 > > > path: > > > logs: /Users/sanyal/Documents/workspace/logs > > > > > By enabling the direct memory buffer, I see out of memory errors on > > > the master node. I don't get the errors if I don't set > > > index.store.fs.memory.enabled=true. > > > > > Am I using it incorrectly, or is this something which should not be > > > used ideally? > > > > > * * * > > > > > [17:51:45,685][INFO][node] [Power, James Dr.] > > > {Elasticsearch/0.8.0}[86452]: Initializing ... > > > [17:51:45,689][INFO][plugins] [Power, James Dr.] > > > Loaded > > > [17:51:47,011][INFO][node] [Power, James Dr.] > > > {Elasticsearch/0.8.0}[86452]: Initialized > > > [17:51:47,012][INFO][node] [Power, James Dr.] > > > {Elasticsearch/0.8.0}[86452]: Starting ... > > > [17:51:47,335][INFO][transport] [Power, James Dr.] > > > bound\_address[inet[/0.0.0.0:9300]], publish\_address[inet[/ > > > 169.254.180.96:9300]] > > > [17:51:50,618][INFO][cluster.service] [Power, James Dr.] > > > New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110] > > > [inet[/169.254.180.96:9300]], Reason: zen-disco- > > > initial\_connect(master) > > > [17:51:50,783][INFO][discovery] [Power, James Dr.] > > > elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110 > > > [17:51:50,890][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_175], cause [gateway], shards [3]/[2], mappings > > > [named-search, stylesheet, view, mm-iphone-alert] > > > [17:51:52,455][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_168], cause [gateway], shards [3]/[2], mappings > > > [video-topic-page, view, video-clip-list] > > > [17:51:53,512][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_164], cause [gateway], shards [3]/[2], mappings > > > [dam-view, video-topic-page, stylesheet, view, video-clip-list] > > > [17:51:54,595][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_165], cause [gateway], shards [3]/[2], mappings > > > [view] > > > [17:51:55,589][INFO][cluster.service] [Power, James Dr.] > > > Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ > > > 169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human > > > Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ > > > 169.254.180.96:9301]]]) > > > [17:51:55,685][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_161], cause [gateway], shards [3]/[2], mappings > > > [view] > > > [17:51:56,872][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_62], cause [gateway], shards [3]/[2], mappings [dam- > > > view, video-topic-page, named-search, stylesheet, view, video-clip- > > > list] > > > [17:52:02,708][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_63], cause [gateway], shards [3]/[2], mappings > > > [view] > > > [17:52:05,479][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_170], cause [gateway], shards [3]/[2], mappings > > > [view] > > > [17:52:07,431][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_14], cause [gateway], shards [3]/[2], mappings > > > [pressbox-asset-list, pressbox-asset, dam-view, audio, named-search, > > > lineup-player, section, vpp-nav-list, article, vpp-promowall, page, > > > video-topic-page, stylesheet, view, video-clip-list, note, lineup, > > > video] > > > [17:52:10,657][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_8], cause [gateway], shards [3]/[2], mappings > > > [video-topic-page, view, video-clip-list] > > > [17:52:13,410][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_9], cause [gateway], shards [3]/[2], mappings > > > [view] > > > [17:52:16,668][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_33], cause [gateway], shards [3]/[2], mappings > > > [video-topic-page, view, video-clip-list] > > > [17:52:20,923][WARN][gateway] [Power, James Dr.] > > > Waited for 30s for indices to be created from the gateway, and not all > > > have been created > > > [17:52:22,477][INFO][http] [Power, James Dr.] > > > bound\_address[inet[/0.0.0.0:9201]], publish\_address[inet[/ > > > 169.254.180.96:9201]] > > > [17:52:23,638][INFO][jmx] [Power, James Dr.] > > > bound\_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi], > > > publish\_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/ > > > jmxrmi] > > > [17:52:24,021][INFO][node] [Power, James Dr.] > > > {Elasticsearch/0.8.0}[86452]: Started > > > [17:52:24,576][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_34], cause [gateway], shards [3]/[2], mappings > > > [video-topic-page, view, video-clip-list] > > > [17:52:30,201][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_43], cause [gateway], shards [3]/[2], mappings > > > [view] > > > [17:52:33,944][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_179], cause [gateway], shards [3]/[2], mappings > > > [named-search, view] > > > [17:52:42,696][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_44], cause [gateway], shards [3]/[2], mappings > > > [view] > > > [17:53:05,119][INFO][cluster.metadata] [Power, James Dr.] > > > Creating Index [hb\_41], cause [gateway], shards [3]/[2], mappings > > > [view] > > > [17:53:54,451][WARN][indices.cluster] [Power, James Dr.] > > > Failed to create shard for index [hb\_41] and shard id [2] > > > org.elasticsearch.util.inject.CreationException: Guice creation > > > errors: > > > > > 1. Error injecting constructor, java.lang.OutOfMemoryError: Direct > > > buffer memory > > > at org.elasticsearch.index.store.fs.NioFsStore.(Unknown > > > Source) > > > while locating org.elasticsearch.index.store.fs.NioFsStore > > > while locating org.elasticsearch.index.store.Store > > > for parameter 2 at > > > org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown > > > Source) > > > while locating > > > org.elasticsearch.index.shard.service.InternalIndexShard > > > while locating org.elasticsearch.index.shard.service.IndexShard > > > Caused by: java.lang.OutOfMemoryError: Direct buffer memory > > > at java.nio.Bits.reserveMemory(Bits.java:633) > > > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95) > > > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) > > > at > > > > > > > org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteB > ufferDirectory.java: > > > > 1. at > > > > > > > org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferD > irectory.java: > > > > 1. at > > > > > > > org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeed > ed(AbstractFsStore.java: > > > > 1. at > > > org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java: > > > 2. at > > > sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown > > > Source) > > > at > > > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru > ctorAccessorImpl.java: > > > > 1. at > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > > at org.elasticsearch.util.inject.DefaultConstructionProxyFactory > > > $1.newInstance(DefaultConstructionProxyFactory.java:53) > > > at > > > > > > > org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje > ctor.java: > > > > 1. at org.elasticsearch.util.inject.ConstructorBindingImpl > > > $Factory.get(ConstructorBindingImpl.java:97) > > > at > > > org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java: > > > 2. at > > > org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter > > > $1.call(ProviderToInternalFactoryAdapter.java:45) > > > at > > org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: > > > > 1. at > > > > > > > org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(Provider > ToInternalFactoryAdapter.java: > > > > 1. at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54) > > > at > > > > > > > org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(Internal > FactoryToProviderAdapter.java: > > > > 1. at > > > > > > > org.elasticsearch.util.inject.SingleParameterInjector.inject(SingleParamete > rInjector.java: > > > > 1. at > > > > > > > org.elasticsearch.util.inject.SingleParameterInjector.getAll(SingleParamete > rInjector.java: > > > > 1. at > > > > > > > org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje > ctor.java: > > > > 1. at org.elasticsearch.util.inject.ConstructorBindingImpl > > > $Factory.get(ConstructorBindingImpl.java:97) > > > at > > > org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java: > > > 2. at > > > org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter > > > $1.call(ProviderToInternalFactoryAdapter.java:45) > > > at > > org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java: > > > > 1. at > > > > > > > org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(Provider > ToInternalFactoryAdapter.java: > > > > 1. at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54) > > > at > > > > ... > > > > read more » ---
**Author:** ![Diptamay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diptamay/32/2805_2.png) [@Diptamay](https://discuss.elastic.co/u/Diptamay)\ **Post date:** [June 28, 2010, 3:14am UTC](https://discuss.elastic.co/t/out-of-memory-if-index-store-fs-memory-enabled-is-true/3043/5 "2010-06-28T03:14:58Z")
Thanks. I shall try that out and let you know. -Diptamay On Jun 26, 7:01 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > The size of the index is not relevant in this case, since the "warm\_cache" > is set to true, each shard will pre-populate a 20mb size of native buffer. > So, in your case, 20mb \* 20 \* 2 (I think you used 2 shards per index) is > 800mb used OOB. You can set the warm\_cache setting to disable it on a per > shard level, or move to master, where 20mb will be pre allocated on the node > level, and not on a per shard level. > > -shay.banon > > On Sun, Jun 27, 2010 at 1:39 AM, diptamay [dipta...@gmail.com](mailto:dipta...@gmail.com) wrote: > > > Hi Shay > > > Yes I created several indices, around 20 or so, since our CMS app > > supports multiple sites and I wanted the indices to be per site. > > However, out of the 50k documents I indexed, majority like 46k or so > > belonged to only 1 index. > > > Thanks for the explanation of the issue (I started both the nodes with > > 3 shards on them.). I could try out the fix by checking out the > > latest, and let you know? > > > By the way, I am using the 64-bit jvm on my snow leopard (64-bit). > > > Thanks > > Diptamay > > > On Jun 26, 3:36 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > My guess is that you create several indices, and the memory store cache > > > is > > > done on a per shard level. I have changed it to be done on a node level, > > > explained here: > > > [Memory Store / FS Memory: Create a node level memory store cache and allocator · Issue #235 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/235). Stil, this > > > is only an assumption, the other thing that I can think of is you using > > > 32 > > > bit JVM? > > > > -shay.banon > > > > On Sat, Jun 26, 2010 at 2:00 AM, diptamay [dipta...@gmail.com](mailto:dipta...@gmail.com) wrote: > > > > > > > Just to see what are the limits, I started 2 nodes (each with default > > > > heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a > > > > pre-created index of like 50k documents, with the following > > > > configuration (in specific I had set > > > > index.store.fs.memory.enabled=true for this run) > > > > > gateway: > > > > type: fs > > > > fs: > > > > location: /Users/sanyal/Documents/workspace/hb\_indices > > > > index: > > > > gateway: > > > > snapshot\_interval : 30s > > > > store: > > > > type: niofs > > > > fs: > > > > memory: > > > > enabled: true > > > > number\_of\_shards : 3 > > > > number\_of\_replicas : 2 > > > > path: > > > > logs: /Users/sanyal/Documents/workspace/logs > > > > > By enabling the direct memory buffer, I see out of memory errors on > > > > the master node. I don't get the errors if I don't set > > > > index.store.fs.memory.enabled=true. > > > > > Am I using it incorrectly, or is this something which should not be > > > > used ideally? > > > > > * * * > > > > > [17:51:45,685][INFO][node] [Power, James Dr.] > > > > {Elasticsearch/0.8.0}[86452]: Initializing ... > > > > [17:51:45,689][INFO][plugins] [Power, James Dr.] > > > > Loaded > > > > [17:51:47,011][INFO][node] [Power, James Dr.] > > > > {Elasticsearch/0.8.0}[86452]: Initialized > > > > [17:51:47,012][INFO][node] [Power, James Dr.] > > > > {Elasticsearch/0.8.0}[86452]: Starting ... > > > > [17:51:47,335][INFO][transport] [Power, James Dr.] > > > > bound\_address[inet[/0.0.0.0:9300]], publish\_address[inet[/ > > > > 169.254.180.96:9300]] > > > > [17:51:50,618][INFO][cluster.service] [Power, James Dr.] > > > > New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110] > > > > [inet[/169.254.180.96:9300]], Reason: zen-disco- > > > > initial\_connect(master) > > > > [17:51:50,783][INFO][discovery] [Power, James Dr.] > > > > elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110 > > > > [17:51:50,890][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_175], cause [gateway], shards [3]/[2], mappings > > > > [named-search, stylesheet, view, mm-iphone-alert] > > > > [17:51:52,455][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_168], cause [gateway], shards [3]/[2], mappings > > > > [video-topic-page, view, video-clip-list] > > > > [17:51:53,512][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_164], cause [gateway], shards [3]/[2], mappings > > > > [dam-view, video-topic-page, stylesheet, view, video-clip-list] > > > > [17:51:54,595][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_165], cause [gateway], shards [3]/[2], mappings > > > > [view] > > > > [17:51:55,589][INFO][cluster.service] [Power, James Dr.] > > > > Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ > > > > 169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human > > > > Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/ > > > > 169.254.180.96:9301]]]) > > > > [17:51:55,685][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_161], cause [gateway], shards [3]/[2], mappings > > > > [view] > > > > [17:51:56,872][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_62], cause [gateway], shards [3]/[2], mappings [dam- > > > > view, video-topic-page, named-search, stylesheet, view, video-clip- > > > > list] > > > > [17:52:02,708][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_63], cause [gateway], shards [3]/[2], mappings > > > > [view] > > > > [17:52:05,479][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_170], cause [gateway], shards [3]/[2], mappings > > > > [view] > > > > [17:52:07,431][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_14], cause [gateway], shards [3]/[2], mappings > > > > [pressbox-asset-list, pressbox-asset, dam-view, audio, named-search, > > > > lineup-player, section, vpp-nav-list, article, vpp-promowall, page, > > > > video-topic-page, stylesheet, view, video-clip-list, note, lineup, > > > > video] > > > > [17:52:10,657][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_8], cause [gateway], shards [3]/[2], mappings > > > > [video-topic-page, view, video-clip-list] > > > > [17:52:13,410][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_9], cause [gateway], shards [3]/[2], mappings > > > > [view] > > > > [17:52:16,668][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_33], cause [gateway], shards [3]/[2], mappings > > > > [video-topic-page, view, video-clip-list] > > > > [17:52:20,923][WARN][gateway] [Power, James Dr.] > > > > Waited for 30s for indices to be created from the gateway, and not all > > > > have been created > > > > [17:52:22,477][INFO][http] [Power, James Dr.] > > > > bound\_address[inet[/0.0.0.0:9201]], publish\_address[inet[/ > > > > 169.254.180.96:9201]] > > > > [17:52:23,638][INFO][jmx] [Power, James Dr.] > > > > bound\_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi], > > > > publish\_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/ > > > > jmxrmi] > > > > [17:52:24,021][INFO][node] [Power, James Dr.] > > > > {Elasticsearch/0.8.0}[86452]: Started > > > > [17:52:24,576][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_34], cause [gateway], shards [3]/[2], mappings > > > > [video-topic-page, view, video-clip-list] > > > > [17:52:30,201][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_43], cause [gateway], shards [3]/[2], mappings > > > > [view] > > > > [17:52:33,944][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_179], cause [gateway], shards [3]/[2], mappings > > > > [named-search, view] > > > > [17:52:42,696][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_44], cause [gateway], shards [3]/[2], mappings > > > > [view] > > > > [17:53:05,119][INFO][cluster.metadata] [Power, James Dr.] > > > > Creating Index [hb\_41], cause [gateway], shards [3]/[2], mappings > > > > [view] > > > > [17:53:54,451][WARN][indices.cluster] [Power, James Dr.] > > > > Failed to create shard for index [hb\_41] and shard id [2] > > > > org.elasticsearch.util.inject.CreationException: Guice creation > > > > errors: > > > > > 1. Error injecting constructor, java.lang.OutOfMemoryError: Direct > > > > buffer memory > > > > at org.elasticsearch.index.store.fs.NioFsStore.(Unknown > > > > Source) > > > > while locating org.elasticsearch.index.store.fs.NioFsStore > > > > while locating org.elasticsearch.index.store.Store > > > > for parameter 2 at > > > > org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown > > > > Source) > > > > while locating > > > > org.elasticsearch.index.shard.service.InternalIndexShard > > > > while locating org.elasticsearch.index.shard.service.IndexShard > > > > Caused by: java.lang.OutOfMemoryError: Direct buffer memory > > > > at java.nio.Bits.reserveMemory(Bits.java:633) > > > > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95) > > > > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) > > > > at > > > org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteB > > ufferDirectory.java: > > > > > > 1. at > > > org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferD > > irectory.java: > > > > > > 1. at > > > org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeed > > ed(AbstractFsStore.java: > > > > > > 1. at > > > > org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java: > > > > 2. at > > > > sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown > > > > Source) > > > > at > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru > > ctorAccessorImpl.java: > > > > > > 1. at > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > > > at org.elasticsearch.util.inject.DefaultConstructionProxyFactory > > > > $1.newInstance(DefaultConstructionProxyFactory.java:53) > > > > at > > > org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje > > ctor.java: > > > > > > 1. > > ... > > read more » ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:22am UTC](https://discuss.elastic.co/t/out-of-memory-if-index-store-fs-memory-enabled-is-true/3043/6 "2017-07-06T04:22:57Z")