Index in memory

Elastic Stack Elasticsearch
9

Thanks Lukas,

I will try mmapfs.

Thanks for all.

Em 17 de abril de 2012 06:01, Lukáš Vlček lukas.vlcek@gmail.com escreveu:

Should have said I tested this on 0.19.0 and 0.19.2.

On Tue, Apr 17, 2012 at 10:58 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Gustavo,

just FYI I switched to in-memory indices as well due to performance
reasons (having slower disk IO can be a real problem). However, I was able
to use index.store.type: mmapfs with local gateway with success (at least it
looks like it is working fine so far).

Shay, is there any issue with using mmapfs indices and local gateway in
multi-node cluster?

Regards,
Lukas

On Tue, Apr 17, 2012 at 9:56 AM, Shay Banon kimchy@gmail.com wrote:

The OS will use the file system cache to boost performance.

On Fri, Apr 13, 2012 at 3:56 PM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

I wanted to make the index in memory. To get better performance.
Would any other way to do this and a similar performance?
If I have 30GB of memory and using FS, the ES will use all the memory
if necessary?

Em 13 de abril de 2012 09:46, Shay Banon kimchy@gmail.com escreveu:

Yea, its a failure that should be fixed, you won't be able to retain
the data when using local gateway and in memory index. So we either need to
delete the index or have it come up empty. Btw, why do you need to have the
index in memory?

On Fri, Apr 13, 2012 at 3:37 PM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Thanks, Shay

When i user store in memory, when i restart the ES i will lost all my
data ?

When i stop and start the ES(0.19.1) i have the massage below.

[2012-04-13 09:32:46,713][INFO ][gateway ] [Death
Adder] recovered [1] indices into cluster_state
[2012-04-13 09:32:46,726][WARN ][indices.cluster ] [Death
Adder] [topicoindex][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[topicoindex][0] shard allocated for local recovery (post api), should
exists, but doesn't
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:107)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
[2012-04-13 09:32:46,756][WARN ][cluster.action.shard ] [Death
Adder] sending failed shard for [topicoindex][0],
node[i1FbZPaYQTSZ663icY_4Ww], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[topicoindex][0] shard
allocated for local recovery (post api), should exists, but doesn't]]]
[2012-04-13 09:32:46,756][WARN ][cluster.action.shard ] [Death
Adder] received shard failed for [topicoindex][0],
node[i1FbZPaYQTSZ663icY_4Ww], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[topicoindex][0] shard
allocated for local recovery (post api), should exists, but doesn't]]]
[2012-04-13 09:32:55,983][WARN ][indices.cluster ] [Death
Adder] [topicoindex][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[topicoindex][0] shard allocated for local recovery (post api), should
exists, but doesn't
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:107)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
[2012-04-13 09:32:56,008][WARN ][cluster.action.shard ] [Death
Adder] sending failed shard for [topicoindex][0],
node[i1FbZPaYQTSZ663icY_4Ww], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[topicoindex][0] shard
allocated for local recovery (post api), should exists, but doesn't]]]
[2012-04-13 09:32:56,008][WARN ][cluster.action.shard ] [Death
Adder] received shard failed for [topicoindex][0],
node[i1FbZPaYQTSZ663icY_4Ww], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[topicoindex][0] shard
allocated for local recovery (post api), should exists, but doesn't]]]

Em 13 de abril de 2012 09:21, Shay Banon kimchy@gmail.com escreveu:

Yes, setting index.store.type to memory will store it in memory.

On Thu, Apr 12, 2012 at 4:45 PM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Hello,

Please, anybody can check is the config below is correct, for use
index in memory.

{
"topicoindex" : {
"settings" : {
"index.analysis.filter.myngram.max_gram" : "200",
"index.analysis.filter.jus_stop.type" : "stop",
"index.store.type" : "memory",
"index.analysis.filter.jus_stop.enable_position_increments" :
"false",
"index.analysis.filter.jus_word_delimiter.type_table_path" :
"/opt/elasticsearch/config/delimiter.txt",
"index.number_of_replicas" : "0",
"index.analysis.filter.jus_word_delimiter.catenate_numbers" :
"true",
"index.analysis.filter.jus_word_delimiter.preserve_original" :
"true",
"index.analysis.filter.jus_word_delimiter.catenate_words" :
"true",
"index.analysis.filter.myngram.min_gram" : "1",
"index.analysis.filter.myngram.type" : "edgeNGram",
"index.analysis.filter.jus_word_delimiter.split_on_numerics" :
"true",
"index.analysis.filter.jus_word_delimiter.catenate_all" :
"true",
"index.analysis.filter.jus_stop.ignore_case" : "true",
"index.analysis.filter.jus_stemmer.name" : "brazilian",
"index.number_of_shards" : "1",
"index.analysis.filter.jus_stop.lang" : "brazilian",
"index.analysis.analyzer.default.filter.3" : "lowercase",
"index.analysis.analyzer.default.filter.4" : "myngram",
"index.analysis.analyzer.default.tokenizer" : "whitespace",
"index.analysis.filter.jus_word_delimiter.type" :
"word_delimiter",
"index.analysis.filter.myngram.side" : "front",
"index.analysis.analyzer.default.filter.1" : "standard",
"index.analysis.analyzer.default.filter.2" :
"jus_word_delimiter",
"index.analysis.filter.my_synonym.synonyms_path" :
"/opt/elasticsearch/config/synonyms_atos.txt",
"index.analysis.analyzer.default.filter.0" : "accentFilter",
"index.analysis.filter.my_synonym.type" : "synonym",
"index.analysis.filter.jus_stemmer.type" : "stemmer",
"index.analysis.filter.jus_stop.stopwords_path" :
"/opt/elasticsearch/config/stopwords.txt",
"index.version.created" : "190199"
}
}
}

Thanks for all.

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia