High availability when creating "in memory" index without persisting the index into a storage

Elastic Stack Elasticsearch
1

Hello!
We have the following scenario:

  1. we would like to maintain an index of documents in memory
  2. we don't want to keep the index persisted in a storage.
    Practically, our index works as temporary index in memory where we add new
    documents and delete the obsoleted ones.
    Under these circumstances, if a node fails, the indexed data contained in it
    will be available on the cluster? Meaning, is the in-memory index replicated
    over the cluster?

We look forward for your answers.

Thank you.
Regards,
Florin

2

Front ending ES with Hazelcast wont be a bad idea.

LINK - http://www.hazelcast.com/

Thanks
Vineeth

On Wed, Oct 5, 2011 at 6:45 PM, Spico Florin spicoflorin@gmail.com wrote:

Hello!
We have the following scenario:

  1. we would like to maintain an index of documents in memory
  2. we don't want to keep the index persisted in a storage.
    Practically, our index works as temporary index in memory where we add new
    documents and delete the obsoleted ones.
    Under these circumstances, if a node fails, the indexed data contained in
    it will be available on the cluster? Meaning, is the in-memory index
    replicated over the cluster?

We look forward for your answers.

Thank you.
Regards,
Florin

3

Well, if you don't need persistence then either Hazelcast or ElasticSearch
will work fine as is. If you want to query by something other than key, then
you would want to include ElasticSearch.

Using ElasticSearch, your data is replicated (if you configure your index
such) whether you use a memory store or file-based storage.

Look at the store types
here: http://www.elasticsearch.org/guide/reference/index-modules/store.html

4

Can you explain in more detail, what do you mean by "front ending".
I have to say that I like both elasticsearch and hazelcast.

On Oct 5, 7:32 pm, Vineeth Mohan vineethmo...@algotree.com wrote:

Front ending ES with Hazelcast wont be a bad idea.

LINK -http://www.hazelcast.com/

Thanks
Vineeth

On Wed, Oct 5, 2011 at 6:45 PM, Spico Florin spicoflo...@gmail.com wrote:

Hello!
We have the following scenario:

  1. we would like to maintain an index of documents in memory
  2. we don't want to keep the index persisted in a storage.
    Practically, our index works as temporary index in memory where we add new
    documents and delete the obsoleted ones.
    Under these circumstances, if a node fails, the indexed data contained in
    it will be available on the cluster? Meaning, is the in-memory index
    replicated over the cluster?

We look forward for your answers.

Thank you.
Regards,
Florin

5

By "front-ending" he was referring to the option of using Hazelcast as your
memcache layer, and having it use ES for persistence. Hazelcast has a simple
MapStore interface you can implement to read/write objects from ES. Since
you don't need persistence, there is little need for that set up.

6

Yes, an in memory index is replicated across the cluster, and if one node
fails, then you have a replica. Note, the internal transaction log used in
elasticsearch is always file based one. I highly suggest just using the file
system based one, its simpler, and you can use the mmapfs one for better
perf. why do you want it to be in memory one?

On Wed, Oct 5, 2011 at 3:15 PM, Spico Florin spicoflorin@gmail.com wrote:

Hello!
We have the following scenario:

  1. we would like to maintain an index of documents in memory
  2. we don't want to keep the index persisted in a storage.
    Practically, our index works as temporary index in memory where we add new
    documents and delete the obsoleted ones.
    Under these circumstances, if a node fails, the indexed data contained in
    it will be available on the cluster? Meaning, is the in-memory index
    replicated over the cluster?

We look forward for your answers.

Thank you.
Regards,
Florin