Out of Memory - if index.store.fs.memory.enabled is true


(Diptamay) #1

Just to see what are the limits, I started 2 nodes (each with default
heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a
pre-created index of like 50k documents, with the following
configuration (in specific I had set
index.store.fs.memory.enabled=true for this run)

gateway:
type: fs
fs:
location: /Users/sanyal/Documents/workspace/hb_indices
index:
gateway:
snapshot_interval : 30s
store:
type: niofs
fs:
memory:
enabled: true
number_of_shards : 3
number_of_replicas : 2
path:
logs: /Users/sanyal/Documents/workspace/logs

By enabling the direct memory buffer, I see out of memory errors on
the master node. I don't get the errors if I don't set
index.store.fs.memory.enabled=true.

Am I using it incorrectly, or is this something which should not be
used ideally?


[17:51:45,685][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initializing ...
[17:51:45,689][INFO ][plugins ] [Power, James Dr.]
Loaded []
[17:51:47,011][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initialized
[17:51:47,012][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Starting ...
[17:51:47,335][INFO ][transport ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9300]], publish_address[inet[/
169.254.180.96:9300]]
[17:51:50,618][INFO ][cluster.service ] [Power, James Dr.]
New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110]
[inet[/169.254.180.96:9300]], Reason: zen-disco-
initial_connect(master)
[17:51:50,783][INFO ][discovery ] [Power, James Dr.]
elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110
[17:51:50,890][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_175], cause [gateway], shards [3]/[2], mappings
[named-search, stylesheet, view, mm-iphone-alert]
[17:51:52,455][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_168], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:51:53,512][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_164], cause [gateway], shards [3]/[2], mappings
[dam-view, video-topic-page, stylesheet, view, video-clip-list]
[17:51:54,595][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_165], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:55,589][INFO ][cluster.service ] [Power, James Dr.]
Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human
Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]]])
[17:51:55,685][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_161], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:56,872][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_62], cause [gateway], shards [3]/[2], mappings [dam-
view, video-topic-page, named-search, stylesheet, view, video-clip-
list]
[17:52:02,708][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_63], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:05,479][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_170], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:07,431][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_14], cause [gateway], shards [3]/[2], mappings
[pressbox-asset-list, pressbox-asset, dam-view, audio, named-search,
lineup-player, section, vpp-nav-list, article, vpp-promowall, page,
video-topic-page, stylesheet, view, video-clip-list, note, lineup,
video]
[17:52:10,657][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_8], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:13,410][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_9], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:16,668][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_33], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:20,923][WARN ][gateway ] [Power, James Dr.]
Waited for 30s for indices to be created from the gateway, and not all
have been created
[17:52:22,477][INFO ][http ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9201]], publish_address[inet[/
169.254.180.96:9201]]
[17:52:23,638][INFO ][jmx ] [Power, James Dr.]
bound_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi],
publish_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/
jmxrmi]
[17:52:24,021][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Started
[17:52:24,576][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_34], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:30,201][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_43], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:33,944][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_179], cause [gateway], shards [3]/[2], mappings
[named-search, view]
[17:52:42,696][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_44], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:05,119][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_41], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:54,451][WARN ][indices.cluster ] [Power, James Dr.]
Failed to create shard for index [hb_41] and shard id [2]
org.elasticsearch.util.inject.CreationException: Guice creation
errors:

  1. Error injecting constructor, java.lang.OutOfMemoryError: Direct
    buffer memory
    at org.elasticsearch.index.store.fs.NioFsStore.(Unknown
    Source)
    while locating org.elasticsearch.index.store.fs.NioFsStore
    while locating org.elasticsearch.index.store.Store
    for parameter 2 at
    org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown
    Source)
    while locating
    org.elasticsearch.index.shard.service.InternalIndexShard
    while locating org.elasticsearch.index.shard.service.IndexShard
    Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:633)
    at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
    at
    org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteBufferDirectory.java:
at

org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferDirectory.java:
110)
at
org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeeded(AbstractFsStore.java:
70)
at org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java:
63)
at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.elasticsearch.util.inject.DefaultConstructionProxyFactory
$1.newInstance(DefaultConstructionProxyFactory.java:53)
at
org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInjector.java:
86)
at org.elasticsearch.util.inject.ConstructorBindingImpl
$Factory.get(ConstructorBindingImpl.java:97)
at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java:
56)
at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter
$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:
791)
at
org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:
42)
at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54)
at
org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:
48)
at
org.elasticsearch.util.inject.SingleParameterInjector.inject(SingleParameterInjector.java:
42)
at
org.elasticsearch.util.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:
66)
at
org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInjector.java:
85)
at org.elasticsearch.util.inject.ConstructorBindingImpl
$Factory.get(ConstructorBindingImpl.java:97)
at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java:
56)
at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter
$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:
791)
at
org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:
42)
at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54)
at
org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:
48)
at org.elasticsearch.util.inject.InjectorBuilder
$1.call(InjectorBuilder.java:197)
at org.elasticsearch.util.inject.InjectorBuilder
$1.call(InjectorBuilder.java:191)
at
org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:
784)
at
org.elasticsearch.util.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:
191)
at
org.elasticsearch.util.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:
173)
at
org.elasticsearch.util.inject.InjectorBuilder.build(InjectorBuilder.java:
110)
at
org.elasticsearch.util.inject.InjectorImpl.createChildInjector(InjectorImpl.java:
124)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:
224)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:
286)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewShards(IndicesClusterStateService.java:
255)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:
112)
at org.elasticsearch.cluster.service.InternalClusterService
$2.run(InternalClusterService.java:198)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)


(Shay Banon) #2

My guess is that you create several indices, and the memory store cache is
done on a per shard level. I have changed it to be done on a node level,
explained here:
http://github.com/elasticsearch/elasticsearch/issues/issue/235. Stil, this
is only an assumption, the other thing that I can think of is you using 32
bit JVM?

-shay.banon

On Sat, Jun 26, 2010 at 2:00 AM, diptamay diptamay@gmail.com wrote:

Just to see what are the limits, I started 2 nodes (each with default
heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a
pre-created index of like 50k documents, with the following
configuration (in specific I had set
index.store.fs.memory.enabled=true for this run)

gateway:
type: fs
fs:
location: /Users/sanyal/Documents/workspace/hb_indices
index:
gateway:
snapshot_interval : 30s
store:
type: niofs
fs:
memory:
enabled: true
number_of_shards : 3
number_of_replicas : 2
path:
logs: /Users/sanyal/Documents/workspace/logs

By enabling the direct memory buffer, I see out of memory errors on
the master node. I don't get the errors if I don't set
index.store.fs.memory.enabled=true.

Am I using it incorrectly, or is this something which should not be
used ideally?


[17:51:45,685][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initializing ...
[17:51:45,689][INFO ][plugins ] [Power, James Dr.]
Loaded []
[17:51:47,011][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initialized
[17:51:47,012][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Starting ...
[17:51:47,335][INFO ][transport ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9300]], publish_address[inet[/
169.254.180.96:9300]]
[17:51:50,618][INFO ][cluster.service ] [Power, James Dr.]
New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110]
[inet[/169.254.180.96:9300]], Reason: zen-disco-
initial_connect(master)
[17:51:50,783][INFO ][discovery ] [Power, James Dr.]
elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110
[17:51:50,890][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_175], cause [gateway], shards [3]/[2], mappings
[named-search, stylesheet, view, mm-iphone-alert]
[17:51:52,455][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_168], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:51:53,512][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_164], cause [gateway], shards [3]/[2], mappings
[dam-view, video-topic-page, stylesheet, view, video-clip-list]
[17:51:54,595][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_165], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:55,589][INFO ][cluster.service ] [Power, James Dr.]
Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human
Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]]])
[17:51:55,685][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_161], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:56,872][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_62], cause [gateway], shards [3]/[2], mappings [dam-
view, video-topic-page, named-search, stylesheet, view, video-clip-
list]
[17:52:02,708][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_63], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:05,479][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_170], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:07,431][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_14], cause [gateway], shards [3]/[2], mappings
[pressbox-asset-list, pressbox-asset, dam-view, audio, named-search,
lineup-player, section, vpp-nav-list, article, vpp-promowall, page,
video-topic-page, stylesheet, view, video-clip-list, note, lineup,
video]
[17:52:10,657][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_8], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:13,410][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_9], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:16,668][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_33], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:20,923][WARN ][gateway ] [Power, James Dr.]
Waited for 30s for indices to be created from the gateway, and not all
have been created
[17:52:22,477][INFO ][http ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9201]], publish_address[inet[/
169.254.180.96:9201]]
[17:52:23,638][INFO ][jmx ] [Power, James Dr.]
bound_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi],
publish_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/
jmxrmi]
[17:52:24,021][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Started
[17:52:24,576][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_34], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:30,201][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_43], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:33,944][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_179], cause [gateway], shards [3]/[2], mappings
[named-search, view]
[17:52:42,696][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_44], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:05,119][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_41], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:54,451][WARN ][indices.cluster ] [Power, James Dr.]
Failed to create shard for index [hb_41] and shard id [2]
org.elasticsearch.util.inject.CreationException: Guice creation
errors:

  1. Error injecting constructor, java.lang.OutOfMemoryError: Direct
    buffer memory
    at org.elasticsearch.index.store.fs.NioFsStore.(Unknown
    Source)
    while locating org.elasticsearch.index.store.fs.NioFsStore
    while locating org.elasticsearch.index.store.Store
    for parameter 2 at
    org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown
    Source)
    while locating
    org.elasticsearch.index.shard.service.InternalIndexShard
    while locating org.elasticsearch.index.shard.service.IndexShard
    Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:633)
    at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
    at

org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteBufferDirectory.java:
247)
at

org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferDirectory.java:
110)
at

org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeeded(AbstractFsStore.java:
70)
at
org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java:
63)
at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
Source)
at

sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.elasticsearch.util.inject.DefaultConstructionProxyFactory
$1.newInstance(DefaultConstructionProxyFactory.java:53)
at

org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInjector.java:
86)
at org.elasticsearch.util.inject.ConstructorBindingImpl
$Factory.get(ConstructorBindingImpl.java:97)
at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java:
56)
at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter
$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:
791)
at

org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:
42)
at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54)
at

org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:
48)
at

org.elasticsearch.util.inject.SingleParameterInjector.inject(SingleParameterInjector.java:
42)
at

org.elasticsearch.util.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:
66)
at

org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInjector.java:
85)
at org.elasticsearch.util.inject.ConstructorBindingImpl
$Factory.get(ConstructorBindingImpl.java:97)
at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java:
56)
at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter
$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:
791)
at

org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:
42)
at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54)
at

org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:
48)
at org.elasticsearch.util.inject.InjectorBuilder
$1.call(InjectorBuilder.java:197)
at org.elasticsearch.util.inject.InjectorBuilder
$1.call(InjectorBuilder.java:191)
at
org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:
784)
at

org.elasticsearch.util.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:
191)
at

org.elasticsearch.util.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:
173)
at
org.elasticsearch.util.inject.InjectorBuilder.build(InjectorBuilder.java:
110)
at

org.elasticsearch.util.inject.InjectorImpl.createChildInjector(InjectorImpl.java:
124)
at

org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:
224)
at

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:
286)
at

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewShards(IndicesClusterStateService.java:
255)
at

org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:
112)
at org.elasticsearch.cluster.service.InternalClusterService
$2.run(InternalClusterService.java:198)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)


(Diptamay) #3

Hi Shay

Yes I created several indices, around 20 or so, since our CMS app
supports multiple sites and I wanted the indices to be per site.
However, out of the 50k documents I indexed, majority like 46k or so
belonged to only 1 index.

Thanks for the explanation of the issue (I started both the nodes with
3 shards on them.). I could try out the fix by checking out the
latest, and let you know?

By the way, I am using the 64-bit jvm on my snow leopard (64-bit).

Thanks
Diptamay

On Jun 26, 3:36 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

My guess is that you create several indices, and the memory store cache is
done on a per shard level. I have changed it to be done on a node level,
explained here:http://github.com/elasticsearch/elasticsearch/issues/issue/235. Stil, this
is only an assumption, the other thing that I can think of is you using 32
bit JVM?

-shay.banon

On Sat, Jun 26, 2010 at 2:00 AM, diptamay dipta...@gmail.com wrote:

Just to see what are the limits, I started 2 nodes (each with default
heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a
pre-created index of like 50k documents, with the following
configuration (in specific I had set
index.store.fs.memory.enabled=true for this run)

gateway:
type: fs
fs:
location: /Users/sanyal/Documents/workspace/hb_indices
index:
gateway:
snapshot_interval : 30s
store:
type: niofs
fs:
memory:
enabled: true
number_of_shards : 3
number_of_replicas : 2
path:
logs: /Users/sanyal/Documents/workspace/logs

By enabling the direct memory buffer, I see out of memory errors on
the master node. I don't get the errors if I don't set
index.store.fs.memory.enabled=true.

Am I using it incorrectly, or is this something which should not be
used ideally?


[17:51:45,685][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initializing ...
[17:51:45,689][INFO ][plugins ] [Power, James Dr.]
Loaded []
[17:51:47,011][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initialized
[17:51:47,012][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Starting ...
[17:51:47,335][INFO ][transport ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9300]], publish_address[inet[/
169.254.180.96:9300]]
[17:51:50,618][INFO ][cluster.service ] [Power, James Dr.]
New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110]
[inet[/169.254.180.96:9300]], Reason: zen-disco-
initial_connect(master)
[17:51:50,783][INFO ][discovery ] [Power, James Dr.]
elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110
[17:51:50,890][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_175], cause [gateway], shards [3]/[2], mappings
[named-search, stylesheet, view, mm-iphone-alert]
[17:51:52,455][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_168], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:51:53,512][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_164], cause [gateway], shards [3]/[2], mappings
[dam-view, video-topic-page, stylesheet, view, video-clip-list]
[17:51:54,595][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_165], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:55,589][INFO ][cluster.service ] [Power, James Dr.]
Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human
Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]]])
[17:51:55,685][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_161], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:56,872][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_62], cause [gateway], shards [3]/[2], mappings [dam-
view, video-topic-page, named-search, stylesheet, view, video-clip-
list]
[17:52:02,708][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_63], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:05,479][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_170], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:07,431][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_14], cause [gateway], shards [3]/[2], mappings
[pressbox-asset-list, pressbox-asset, dam-view, audio, named-search,
lineup-player, section, vpp-nav-list, article, vpp-promowall, page,
video-topic-page, stylesheet, view, video-clip-list, note, lineup,
video]
[17:52:10,657][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_8], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:13,410][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_9], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:16,668][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_33], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:20,923][WARN ][gateway ] [Power, James Dr.]
Waited for 30s for indices to be created from the gateway, and not all
have been created
[17:52:22,477][INFO ][http ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9201]], publish_address[inet[/
169.254.180.96:9201]]
[17:52:23,638][INFO ][jmx ] [Power, James Dr.]
bound_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi],
publish_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/
jmxrmi]
[17:52:24,021][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Started
[17:52:24,576][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_34], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:30,201][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_43], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:33,944][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_179], cause [gateway], shards [3]/[2], mappings
[named-search, view]
[17:52:42,696][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_44], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:05,119][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_41], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:54,451][WARN ][indices.cluster ] [Power, James Dr.]
Failed to create shard for index [hb_41] and shard id [2]
org.elasticsearch.util.inject.CreationException: Guice creation
errors:

  1. Error injecting constructor, java.lang.OutOfMemoryError: Direct
    buffer memory
    at org.elasticsearch.index.store.fs.NioFsStore.(Unknown
    Source)
    while locating org.elasticsearch.index.store.fs.NioFsStore
    while locating org.elasticsearch.index.store.Store
    for parameter 2 at
    org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown
    Source)
    while locating
    org.elasticsearch.index.shard.service.InternalIndexShard
    while locating org.elasticsearch.index.shard.service.IndexShard
    Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:633)
    at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
    at

org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteB ufferDirectory.java:
247)
at

org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferD irectory.java:
110)
at

org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeed ed(AbstractFsStore.java:
70)
at
org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java:
63)
at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
Source)
at

sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru ctorAccessorImpl.java:
27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.elasticsearch.util.inject.DefaultConstructionProxyFactory
$1.newInstance(DefaultConstructionProxyFactory.java:53)
at

org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje ctor.java:
86)
at org.elasticsearch.util.inject.ConstructorBindingImpl
$Factory.get(ConstructorBindingImpl.java:97)
at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java:
56)
at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter
$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:
791)
at

org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(Provider ToInternalFactoryAdapter.java:
42)
at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54)
at

org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(Internal FactoryToProviderAdapter.java:
48)
at

org.elasticsearch.util.inject.SingleParameterInjector.inject(SingleParamete rInjector.java:
42)
at

org.elasticsearch.util.inject.SingleParameterInjector.getAll(SingleParamete rInjector.java:
66)
at

org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje ctor.java:
85)
at org.elasticsearch.util.inject.ConstructorBindingImpl
$Factory.get(ConstructorBindingImpl.java:97)
at org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java:
56)
at org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter
$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:
791)
at

org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(Provider ToInternalFactoryAdapter.java:
42)
at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54)
at

...

read more »


(Shay Banon) #4

The size of the index is not relevant in this case, since the "warm_cache"
is set to true, each shard will pre-populate a 20mb size of native buffer.
So, in your case, 20mb * 20 * 2 (I think you used 2 shards per index) is
800mb used OOB. You can set the warm_cache setting to disable it on a per
shard level, or move to master, where 20mb will be pre allocated on the node
level, and not on a per shard level.

-shay.banon

On Sun, Jun 27, 2010 at 1:39 AM, diptamay diptamay@gmail.com wrote:

Hi Shay

Yes I created several indices, around 20 or so, since our CMS app
supports multiple sites and I wanted the indices to be per site.
However, out of the 50k documents I indexed, majority like 46k or so
belonged to only 1 index.

Thanks for the explanation of the issue (I started both the nodes with
3 shards on them.). I could try out the fix by checking out the
latest, and let you know?

By the way, I am using the 64-bit jvm on my snow leopard (64-bit).

Thanks
Diptamay

On Jun 26, 3:36 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

My guess is that you create several indices, and the memory store cache
is
done on a per shard level. I have changed it to be done on a node level,
explained here:
http://github.com/elasticsearch/elasticsearch/issues/issue/235. Stil, this
is only an assumption, the other thing that I can think of is you using
32
bit JVM?

-shay.banon

On Sat, Jun 26, 2010 at 2:00 AM, diptamay dipta...@gmail.com wrote:

Just to see what are the limits, I started 2 nodes (each with default
heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a
pre-created index of like 50k documents, with the following
configuration (in specific I had set
index.store.fs.memory.enabled=true for this run)

gateway:
type: fs
fs:
location: /Users/sanyal/Documents/workspace/hb_indices
index:
gateway:
snapshot_interval : 30s
store:
type: niofs
fs:
memory:
enabled: true
number_of_shards : 3
number_of_replicas : 2
path:
logs: /Users/sanyal/Documents/workspace/logs

By enabling the direct memory buffer, I see out of memory errors on
the master node. I don't get the errors if I don't set
index.store.fs.memory.enabled=true.

Am I using it incorrectly, or is this something which should not be
used ideally?


[17:51:45,685][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initializing ...
[17:51:45,689][INFO ][plugins ] [Power, James Dr.]
Loaded []
[17:51:47,011][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initialized
[17:51:47,012][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Starting ...
[17:51:47,335][INFO ][transport ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9300]], publish_address[inet[/
169.254.180.96:9300]]
[17:51:50,618][INFO ][cluster.service ] [Power, James Dr.]
New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110]
[inet[/169.254.180.96:9300]], Reason: zen-disco-
initial_connect(master)
[17:51:50,783][INFO ][discovery ] [Power, James Dr.]
elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110
[17:51:50,890][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_175], cause [gateway], shards [3]/[2], mappings
[named-search, stylesheet, view, mm-iphone-alert]
[17:51:52,455][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_168], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:51:53,512][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_164], cause [gateway], shards [3]/[2], mappings
[dam-view, video-topic-page, stylesheet, view, video-clip-list]
[17:51:54,595][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_165], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:55,589][INFO ][cluster.service ] [Power, James Dr.]
Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human
Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]]])
[17:51:55,685][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_161], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:56,872][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_62], cause [gateway], shards [3]/[2], mappings [dam-
view, video-topic-page, named-search, stylesheet, view, video-clip-
list]
[17:52:02,708][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_63], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:05,479][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_170], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:07,431][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_14], cause [gateway], shards [3]/[2], mappings
[pressbox-asset-list, pressbox-asset, dam-view, audio, named-search,
lineup-player, section, vpp-nav-list, article, vpp-promowall, page,
video-topic-page, stylesheet, view, video-clip-list, note, lineup,
video]
[17:52:10,657][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_8], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:13,410][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_9], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:16,668][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_33], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:20,923][WARN ][gateway ] [Power, James Dr.]
Waited for 30s for indices to be created from the gateway, and not all
have been created
[17:52:22,477][INFO ][http ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9201]], publish_address[inet[/
169.254.180.96:9201]]
[17:52:23,638][INFO ][jmx ] [Power, James Dr.]
bound_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi],
publish_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/
jmxrmi]
[17:52:24,021][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Started
[17:52:24,576][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_34], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:30,201][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_43], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:33,944][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_179], cause [gateway], shards [3]/[2], mappings
[named-search, view]
[17:52:42,696][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_44], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:05,119][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_41], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:54,451][WARN ][indices.cluster ] [Power, James Dr.]
Failed to create shard for index [hb_41] and shard id [2]
org.elasticsearch.util.inject.CreationException: Guice creation
errors:

  1. Error injecting constructor, java.lang.OutOfMemoryError: Direct
    buffer memory
    at org.elasticsearch.index.store.fs.NioFsStore.(Unknown
    Source)
    while locating org.elasticsearch.index.store.fs.NioFsStore
    while locating org.elasticsearch.index.store.Store
    for parameter 2 at
    org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown
    Source)
    while locating
    org.elasticsearch.index.shard.service.InternalIndexShard
    while locating org.elasticsearch.index.shard.service.IndexShard
    Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:633)
    at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
    at

org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteB
ufferDirectory.java:

  1. at

org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferD
irectory.java:

  1. at

org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeed
ed(AbstractFsStore.java:

  1. at
    org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java:
  2. at
    sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown

Source)
at

sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru
ctorAccessorImpl.java:

  1. at
    java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at org.elasticsearch.util.inject.DefaultConstructionProxyFactory

$1.newInstance(DefaultConstructionProxyFactory.java:53)
at

org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje
ctor.java:

  1. at org.elasticsearch.util.inject.ConstructorBindingImpl
    $Factory.get(ConstructorBindingImpl.java:97)
    at
    org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java:
  1. at
    org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter

$1.call(ProviderToInternalFactoryAdapter.java:45)
at

org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:

  1. at

org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(Provider
ToInternalFactoryAdapter.java:

  1. at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54)
    at

org.elasticsearch.util.inject.InternalFactoryToProviderAdapter.get(Internal
FactoryToProviderAdapter.java:

  1. at

org.elasticsearch.util.inject.SingleParameterInjector.inject(SingleParamete
rInjector.java:

  1. at

org.elasticsearch.util.inject.SingleParameterInjector.getAll(SingleParamete
rInjector.java:

  1. at

org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje
ctor.java:

  1. at org.elasticsearch.util.inject.ConstructorBindingImpl
    $Factory.get(ConstructorBindingImpl.java:97)
    at
    org.elasticsearch.util.inject.FactoryProxy.get(FactoryProxy.java:
  1. at
    org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter

$1.call(ProviderToInternalFactoryAdapter.java:45)
at

org.elasticsearch.util.inject.InjectorImpl.callInContext(InjectorImpl.java:

  1. at

org.elasticsearch.util.inject.ProviderToInternalFactoryAdapter.get(Provider
ToInternalFactoryAdapter.java:

  1. at org.elasticsearch.util.inject.Scopes$1$1.get(Scopes.java:54)
    at

...

read more »


(Diptamay) #5

Thanks. I shall try that out and let you know.

-Diptamay

On Jun 26, 7:01 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

The size of the index is not relevant in this case, since the "warm_cache"
is set to true, each shard will pre-populate a 20mb size of native buffer.
So, in your case, 20mb * 20 * 2 (I think you used 2 shards per index) is
800mb used OOB. You can set the warm_cache setting to disable it on a per
shard level, or move to master, where 20mb will be pre allocated on the node
level, and not on a per shard level.

-shay.banon

On Sun, Jun 27, 2010 at 1:39 AM, diptamay dipta...@gmail.com wrote:

Hi Shay

Yes I created several indices, around 20 or so, since our CMS app
supports multiple sites and I wanted the indices to be per site.
However, out of the 50k documents I indexed, majority like 46k or so
belonged to only 1 index.

Thanks for the explanation of the issue (I started both the nodes with
3 shards on them.). I could try out the fix by checking out the
latest, and let you know?

By the way, I am using the 64-bit jvm on my snow leopard (64-bit).

Thanks
Diptamay

On Jun 26, 3:36 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

My guess is that you create several indices, and the memory store cache
is
done on a per shard level. I have changed it to be done on a node level,
explained here:
http://github.com/elasticsearch/elasticsearch/issues/issue/235. Stil, this
is only an assumption, the other thing that I can think of is you using
32
bit JVM?

-shay.banon

On Sat, Jun 26, 2010 at 2:00 AM, diptamay dipta...@gmail.com wrote:

Just to see what are the limits, I started 2 nodes (each with default
heap setting of 256 MB - 1 GB) on my laptop (mac 2.26 GHz, 4 GB), on a
pre-created index of like 50k documents, with the following
configuration (in specific I had set
index.store.fs.memory.enabled=true for this run)

gateway:
type: fs
fs:
location: /Users/sanyal/Documents/workspace/hb_indices
index:
gateway:
snapshot_interval : 30s
store:
type: niofs
fs:
memory:
enabled: true
number_of_shards : 3
number_of_replicas : 2
path:
logs: /Users/sanyal/Documents/workspace/logs

By enabling the direct memory buffer, I see out of memory errors on
the master node. I don't get the errors if I don't set
index.store.fs.memory.enabled=true.

Am I using it incorrectly, or is this something which should not be
used ideally?


[17:51:45,685][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initializing ...
[17:51:45,689][INFO ][plugins ] [Power, James Dr.]
Loaded []
[17:51:47,011][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Initialized
[17:51:47,012][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Starting ...
[17:51:47,335][INFO ][transport ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9300]], publish_address[inet[/
169.254.180.96:9300]]
[17:51:50,618][INFO ][cluster.service ] [Power, James Dr.]
New Master [Power, James Dr.][bf6c54dc-e250-4fd5-90ce-1a79c3e56110]
[inet[/169.254.180.96:9300]], Reason: zen-disco-
initial_connect(master)
[17:51:50,783][INFO ][discovery ] [Power, James Dr.]
elasticsearch/bf6c54dc-e250-4fd5-90ce-1a79c3e56110
[17:51:50,890][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_175], cause [gateway], shards [3]/[2], mappings
[named-search, stylesheet, view, mm-iphone-alert]
[17:51:52,455][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_168], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:51:53,512][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_164], cause [gateway], shards [3]/[2], mappings
[dam-view, video-topic-page, stylesheet, view, video-clip-list]
[17:51:54,595][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_165], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:55,589][INFO ][cluster.service ] [Power, James Dr.]
Added {[Human Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]],}, Reason: zen-disco-receive(from node[[Human
Torch II][5cb03c42-56e4-4561-93dc-3c7cf52dc414][inet[/
169.254.180.96:9301]]])
[17:51:55,685][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_161], cause [gateway], shards [3]/[2], mappings
[view]
[17:51:56,872][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_62], cause [gateway], shards [3]/[2], mappings [dam-
view, video-topic-page, named-search, stylesheet, view, video-clip-
list]
[17:52:02,708][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_63], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:05,479][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_170], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:07,431][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_14], cause [gateway], shards [3]/[2], mappings
[pressbox-asset-list, pressbox-asset, dam-view, audio, named-search,
lineup-player, section, vpp-nav-list, article, vpp-promowall, page,
video-topic-page, stylesheet, view, video-clip-list, note, lineup,
video]
[17:52:10,657][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_8], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:13,410][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_9], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:16,668][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_33], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:20,923][WARN ][gateway ] [Power, James Dr.]
Waited for 30s for indices to be created from the gateway, and not all
have been created
[17:52:22,477][INFO ][http ] [Power, James Dr.]
bound_address[inet[/0.0.0.0:9201]], publish_address[inet[/
169.254.180.96:9201]]
[17:52:23,638][INFO ][jmx ] [Power, James Dr.]
bound_address[service:jmx:rmi:///jndi/rmi://:9401/jmxrmi],
publish_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9401/
jmxrmi]
[17:52:24,021][INFO ][node ] [Power, James Dr.]
{ElasticSearch/0.8.0}[86452]: Started
[17:52:24,576][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_34], cause [gateway], shards [3]/[2], mappings
[video-topic-page, view, video-clip-list]
[17:52:30,201][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_43], cause [gateway], shards [3]/[2], mappings
[view]
[17:52:33,944][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_179], cause [gateway], shards [3]/[2], mappings
[named-search, view]
[17:52:42,696][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_44], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:05,119][INFO ][cluster.metadata ] [Power, James Dr.]
Creating Index [hb_41], cause [gateway], shards [3]/[2], mappings
[view]
[17:53:54,451][WARN ][indices.cluster ] [Power, James Dr.]
Failed to create shard for index [hb_41] and shard id [2]
org.elasticsearch.util.inject.CreationException: Guice creation
errors:

  1. Error injecting constructor, java.lang.OutOfMemoryError: Direct
    buffer memory
    at org.elasticsearch.index.store.fs.NioFsStore.(Unknown
    Source)
    while locating org.elasticsearch.index.store.fs.NioFsStore
    while locating org.elasticsearch.index.store.Store
    for parameter 2 at
    org.elasticsearch.index.shard.service.InternalIndexShard.(Unknown
    Source)
    while locating
    org.elasticsearch.index.shard.service.InternalIndexShard
    while locating org.elasticsearch.index.shard.service.IndexShard
    Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:633)
    at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
    at

org.elasticsearch.index.store.memory.ByteBufferDirectory.createBuffer(ByteB
ufferDirectory.java:

  1. at

org.elasticsearch.index.store.memory.ByteBufferDirectory.(ByteBufferD
irectory.java:

  1. at

org.elasticsearch.index.store.fs.AbstractFsStore.buildSwitchDirectoryIfNeed
ed(AbstractFsStore.java:

  1. at
    org.elasticsearch.index.store.fs.NioFsStore.(NioFsStore.java:
  2. at
    sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown

Source)
at

sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru
ctorAccessorImpl.java:

  1. at
    java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at org.elasticsearch.util.inject.DefaultConstructionProxyFactory

$1.newInstance(DefaultConstructionProxyFactory.java:53)
at

org.elasticsearch.util.inject.ConstructorInjector.construct(ConstructorInje
ctor.java:

...

read more »


(system) #6