We are replacing a fairly flat mysql database with elasticsearch. One of
the things we store currently is a time-series of scores. Lots and lots of
rows for a given foreign key. One thing I like about elasticsearch is that
versioning is baked in. The current thinking is to use versioning for
storing the timeseries of scores.
So instead of many rows of scores for a given object in mysql, we would
have 1 score document in elasticsearch. Each score change would be a
different version. This would however lead to 10s of not 100s of thousands
of versions for a given document.
At what point should we be worried about elasticsearch's ability to
reliably quickly go through versions? Is there a capability in the API to
request a range of versions for a given document?
At what point should we be worried about elasticsearch's ability to
reliably quickly go through versions? Is there a capability in the API
to request a range of versions for a given document?
There is only a single version of the document, with the current version
number.
More context: I think Elasticsearch's versioning is really only for optimistic
concurrency controlhttp://www.elasticsearch.org/blog/2011/02/08/versioning.html.
It does not persist old versions of a document; its concept of "versioning"
is merely a number that gets updated when the document is overwritten.
You could still use Elasticsearch to maintain a time series, but not using
their built-in versioning feature. Take a peek at http://stackoverflow.com/questions/8218309/can-we-retrieve-previous-source-docs-with-elastic-search-versions
for some approaches. Your use case may be simpler still, e.g. you could
store the "timestamp" as a field and do range queries over it, but keep in
mind you'd have the extra overhead of duplicating the rest of the
documents' fields for each timestamp.
On Monday, February 11, 2013 7:59:07 AM UTC-8, Clinton Gormley wrote:
At what point should we be worried about elasticsearch's ability to
reliably quickly go through versions? Is there a capability in the API
to request a range of versions for a given document?
There is only a single version of the document, with the current version
number.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.