ES as persistent client (desktop) cache

Elastic Stack Elasticsearch
1

Currently I use in a rich client application SQLite as a key-value store
for persistent caching on the client machines (Windows desktops). In
addition I've implemented a quite simple inverted index to speed up
searching of the data within the client process.(There're several hundred
thousand data objects used during a session.)

Now I need to build a similar application using a different technology (now
.NET). I wonder if it is a good idea to use ES (no shards, no replicas,
simplest setup) for both, persistent caching and fast searches on the
client desktop machine. From the functional point of view it's very
promising. All data objects that are loaded to the client process are also
available in the local ES instance. To perform a search in the client
process I would query the local ES instance but limit the search to the
data objects which are in the current view of the client process. This
could be done by maintaining a helper document in ES, where I put the ids
of the relevant data objects for each view.

I'm not very concerned about deployment, but more about relaiability and
maintainance? ES would be started and stopped by the leading .NET app, so
it would be restartet frequently.

Any comments or opinions are highly appreciated.

Cheers,

Jan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi,

Am 23.02.13 22:03, schrieb knacktus@googlemail.com:

Currently I use in a rich client application SQLite as a key-value
store for persistent caching on the client machines (Windows
desktops). In addition I've implemented a quite simple inverted index
to speed up searching of the data within the client process.(There're
several hundred thousand data objects used during a session.)

Now I need to build a similar application using a different technology
(now .NET). I wonder if it is a good idea to use ES (no shards, no
replicas, simplest setup) for both, persistent caching and fast
searches on the client desktop machine. From the functional point of
view it's very promising. All data objects that are loaded to the
client process are also available in the local ES instance. To perform
a search in the client process I would query the local ES instance but
limit the search to the data objects which are in the current view of
the client process. This could be done by maintaining a helper
document in ES, where I put the ids of the relevant data objects for
each view.

You won't need to create a "helper document". To model views, you should
consider index types.

I'm not very concerned about deployment, but more about relaiability
and maintainance? ES would be started and stopped by the leading .NET
app, so it would be restartet frequently.

Do you suggest ES is not reliable or maintainable? No, it's reliable
and maintainable, but, as every piece of software, it's not feature
complete (see the missing backup/restore tool which is work in
progress). It's up to your decision if you can live with that.

Note, Elasticsearch can be used to "simulate" a key/value store with
opaque values. Just

  • disable indexing completely
  • disable _source
  • disable _all
  • disable dynamic mapping
  • use the document _id for the key
  • put up a single field "value" with "index: no" and ask in all document
    HTTP GET for this field (fields=value)

This means, disable index and search completey. If you want to confirm
if this is the strength of ES, I must tell, no, it's not, the strength
is enabling indexing and searching.

Basically you can force almost every database or document-oriented
system to behave like a key/value-store, if it's SQLite, ES, MongoDB,
CouchDB, Solr, Lucene, MySQL, Oracle ... if this fits to your
requirements, why not, run your benchmarks and find out if you are
satisfied with the results. But they are not BerkeleyDB, Redis, or
Memcached. Note, the simplest key/value-store still is the file system.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks Jörg.

My concerns regarding reliability and maintainability are solely in the
context of a "left alone" installation on the user's client machines (I
know, I didn't make that clear ... :-)). But I feel more comfortable the
more I've read about the monitoring API. Also, I can rebuild the cache from
scratch if something goes wrong (as it is only a client-side cache).

I guess the main task is to develop suitable initialisation and monitoring
routines to perform tasks like

  • finding sensible memory settings based on the available RAM
  • monitoring CPU und disk space usage
  • initialise countermeasures if indicated - this could be a complete
    reset during the runtime of the main applications process ...

Cheers,

Jan
On Sunday, February 24, 2013 10:38:22 AM UTC+13, Jörg Prante wrote:

Hi,

Am 23.02.13 22:03, schrieb knac...@googlemail.com <javascript:>:

Currently I use in a rich client application SQLite as a key-value
store for persistent caching on the client machines (Windows
desktops). In addition I've implemented a quite simple inverted index
to speed up searching of the data within the client process.(There're
several hundred thousand data objects used during a session.)

Now I need to build a similar application using a different technology
(now .NET). I wonder if it is a good idea to use ES (no shards, no
replicas, simplest setup) for both, persistent caching and fast
searches on the client desktop machine. From the functional point of
view it's very promising. All data objects that are loaded to the
client process are also available in the local ES instance. To perform
a search in the client process I would query the local ES instance but
limit the search to the data objects which are in the current view of
the client process. This could be done by maintaining a helper
document in ES, where I put the ids of the relevant data objects for
each view.

You won't need to create a "helper document". To model views, you should
consider index types.

I'm not very concerned about deployment, but more about relaiability
and maintainance? ES would be started and stopped by the leading .NET
app, so it would be restartet frequently.

Do you suggest ES is not reliable or maintainable? No, it's reliable
and maintainable, but, as every piece of software, it's not feature
complete (see the missing backup/restore tool which is work in
progress). It's up to your decision if you can live with that.

Note, Elasticsearch can be used to "simulate" a key/value store with
opaque values. Just

  • disable indexing completely
  • disable _source
  • disable _all
  • disable dynamic mapping
  • use the document _id for the key
  • put up a single field "value" with "index: no" and ask in all document
    HTTP GET for this field (fields=value)

This means, disable index and search completey. If you want to confirm
if this is the strength of ES, I must tell, no, it's not, the strength
is enabling indexing and searching.

Basically you can force almost every database or document-oriented
system to behave like a key/value-store, if it's SQLite, ES, MongoDB,
CouchDB, Solr, Lucene, MySQL, Oracle ... if this fits to your
requirements, why not, run your benchmarks and find out if you are
satisfied with the results. But they are not BerkeleyDB, Redis, or
Memcached. Note, the simplest key/value-store still is the file system.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.