How to switch from file storage to memory storage?

Elastic Stack Elasticsearch
1

I wanted to run some performance tests against in memory indexes using
the index module store. I set up my elasticsearch.yaml file similar to
what is in this thread:
http://elasticsearch-users.115913.n3.nabble.com/Out-of-Memory-if-index-store-fs-memory-enabled-is-true-td923306.html#a923306

So, the index portion of my config looks like this:
index:
analysis :
analyzer :
verity_tokenizer :
type: pattern
lowercase: true
pattern: '(?:(?!\w).)+'
stopwords: none
store:
type: niofs
fs:
memory:
enabled: true

However, after restarting the cluster, it doesn't appear the setting
is taking effect. I've tried playing around with these settings, but
can't get anything to stick.

Any idea what I am missing?

Thanks,
Paul

2

I was actually able to switch from disk to memory. I have about 14M
records in my Gateway. When I started up brand new Elasticsearch
servers, I could watch the memory usage on each server go up.

Going through the docs and also my configuration. I'm wondering if you
need to change "type: niofs" to "type: memory"

On Aug 26, 8:49 am, Paul ppea...@gmail.com wrote:

I wanted to run some performance tests against in memory indexes using
the index module store. I set up my elasticsearch.yaml file similar to
what is in this thread:http://elasticsearch-users.115913.n3.nabble.com/Out-of-Memory-if-inde...

So, the index portion of my config looks like this:
index:
analysis :
analyzer :
verity_tokenizer :
type: pattern
lowercase: true
pattern: '(?:(?!\w).)+'
stopwords: none
store:
type: niofs
fs:
memory:
enabled: true

However, after restarting the cluster, it doesn't appear the setting
is taking effect. I've tried playing around with these settings, but
can't get anything to stick.

Any idea what I am missing?

Thanks,
Paul

3

Nice, thanks! Seems obvious in retrospect.

On Aug 26, 1:20 pm, David Jensen djense...@gmail.com wrote:

I was actually able to switch from disk to memory. I have about 14M
records in my Gateway. When I started up brand new Elasticsearch
servers, I could watch the memory usage on each server go up.

Going through the docs and also my configuration. I'm wondering if you
need to change "type: niofs" to "type: memory"

On Aug 26, 8:49 am, Paul ppea...@gmail.com wrote:

I wanted to run some performance tests against in memory indexes using
the index module store. I set up my elasticsearch.yaml file similar to
what is in this thread:http://elasticsearch-users.115913.n3.nabble.com/Out-of-Memory-if-inde...

So, the index portion of my config looks like this:
index:
analysis :
analyzer :
verity_tokenizer :
type: pattern
lowercase: true
pattern: '(?:(?!\w).)+'
stopwords: none
store:
type: niofs
fs:
memory:
enabled: true

However, after restarting the cluster, it doesn't appear the setting
is taking effect. I've tried playing around with these settings, but
can't get anything to stick.

Any idea what I am missing?

Thanks,
Paul

4

So, my config file for all indexes in memory looks like:
index:
store:
type: memory
fs:
memory:
enabled: true

This results in all indexes going to memory. I am looking to go with a
hybrid approach, some in memory and some out of memory.

I believe I should be able to accomplish this by hacking the metadata
file to include this setting, eg:
"settings" : {
"index.number_of_replicas" : "0",
"index.number_of_shards" : "2",
"index.store.fs.memory.enabled" : "true",
"index.mapper.dynamic" : "false"
},

However, this doesn't seem to work. I also tried out the index
settings update API(GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine
issues/302), but it doesn't appear to support this option, yet.

Any ideas on how to update a specific index to go straight to memory?

This is probably all a moot point, but I wanted to run performance
tests to prove it, as I believe that the OS's disk cache should be
able to perform similarly to a true in memory index, as long as,
enough RAM is available. I'd be happy to be proven wrong about that,
though.

Thanks,
Paul

On Aug 26, 1:54 pm, Paul ppea...@gmail.com wrote:

Nice, thanks! Seems obvious in retrospect.

On Aug 26, 1:20 pm, David Jensen djense...@gmail.com wrote:

I was actually able to switch from disk to memory. I have about 14M
records in my Gateway. When I started up brand new Elasticsearch
servers, I could watch the memory usage on each server go up.

Going through the docs and also my configuration. I'm wondering if you
need to change "type: niofs" to "type: memory"

On Aug 26, 8:49 am, Paul ppea...@gmail.com wrote:

I wanted to run some performance tests against in memory indexes using
the index module store. I set up my elasticsearch.yaml file similar to
what is in this thread:http://elasticsearch-users.115913.n3.nabble.com/Out-of-Memory-if-inde...

So, the index portion of my config looks like this:
index:
analysis :
analyzer :
verity_tokenizer :
type: pattern
lowercase: true
pattern: '(?:(?!\w).)+'
stopwords: none
store:
type: niofs
fs:
memory:
enabled: true

However, after restarting the cluster, it doesn't appear the setting
is taking effect. I've tried playing around with these settings, but
can't get anything to stick.

Any idea what I am missing?

Thanks,
Paul

5

Hi,

Do you want to change it on the gateway side? When you create an index,
you can specific the settings for it, including if it will use memory index
or not. If you want to go and hack the metadata at the gateway side, you can
shutdown the cluster, add a setting (for the indices you want) -
"index.store.type" : "memory".

-shay.banon

On Fri, Aug 27, 2010 at 12:42 AM, Paul ppearcy@gmail.com wrote:

So, my config file for all indexes in memory looks like:
index:
store:
type: memory
fs:
memory:
enabled: true

This results in all indexes going to memory. I am looking to go with a
hybrid approach, some in memory and some out of memory.

I believe I should be able to accomplish this by hacking the metadata
file to include this setting, eg:
"settings" : {
"index.number_of_replicas" : "0",
"index.number_of_shards" : "2",
"index.store.fs.memory.enabled" : "true",
"index.mapper.dynamic" : "false"
},

However, this doesn't seem to work. I also tried out the index
settings update API(GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine
issues/302), but it doesn't appear to support this option, yet.

Any ideas on how to update a specific index to go straight to memory?

This is probably all a moot point, but I wanted to run performance
tests to prove it, as I believe that the OS's disk cache should be
able to perform similarly to a true in memory index, as long as,
enough RAM is available. I'd be happy to be proven wrong about that,
though.

Thanks,
Paul

On Aug 26, 1:54 pm, Paul ppea...@gmail.com wrote:

Nice, thanks! Seems obvious in retrospect.

On Aug 26, 1:20 pm, David Jensen djense...@gmail.com wrote:

I was actually able to switch from disk to memory. I have about 14M
records in my Gateway. When I started up brand new Elasticsearch
servers, I could watch the memory usage on each server go up.

Going through the docs and also my configuration. I'm wondering if you
need to change "type: niofs" to "type: memory"

On Aug 26, 8:49 am, Paul ppea...@gmail.com wrote:

I wanted to run some performance tests against in memory indexes
using
the index module store. I set up my elasticsearch.yaml file similar
to
what is in this thread:
http://elasticsearch-users.115913.n3.nabble.com/Out-of-Memory-if-inde...

So, the index portion of my config looks like this:
index:
analysis :
analyzer :
verity_tokenizer :
type: pattern
lowercase: true
pattern: '(?:(?!\w).)+'
stopwords: none
store:
type: niofs
fs:
memory:
enabled: true

However, after restarting the cluster, it doesn't appear the setting
is taking effect. I've tried playing around with these settings, but
can't get anything to stick.

Any idea what I am missing?

Thanks,
Paul