Get consistency

Elastic Stack Elasticsearch
1

Hi Shay,

the current model we have for our data allows partial Documents to be
indexed. At some point in the future we may then add the missing fields.
Currently I do this like so (in pseudo):

function update(PartialDoc update) {
PartialDoc doc = get();
doc.merge(update);
index(doc);
}

The problem is that the initial indexing of the doc may not have happened
before we call get so after the merge+index,* *we have lost the
previously indexed fields.

Is there a way to specify a consistency level in ElasticSearch (like
Cassandra), or couls Elastic Search internally provide update functionality?

Thanks,

Paul

--

Paul Loy
paul@keteracel.com
http://www.keteracel.com/paul

2

Hi Paul,

It sounds like you have a similar requirement to us. After a brief
discussion with Shay, I've created the following issue on github

It is not clear what the best way to handle this requirement is. Having get
method work in real-time may be one potential solution. There may be better
ways to solve this as well.

Regards,
Berkay Mollamustafaoglu
http://www.ifountain.com
mberkay on yahoo, google and skype

On Thu, Jun 3, 2010 at 12:11 PM, Paul Loy keteracel@gmail.com wrote:

Hi Shay,

the current model we have for our data allows partial Documents to be
indexed. At some point in the future we may then add the missing fields.
Currently I do this like so (in pseudo):

function update(PartialDoc update) {
PartialDoc doc = get();
doc.merge(update);
index(doc);
}

The problem is that the initial indexing of the doc may not have happened
before we call get so after the merge+index,* *we have lost the
previously indexed fields.

Is there a way to specify a consistency level in Elasticsearch (like
Cassandra), or couls Elastic Search internally provide update functionality?

Thanks,

Paul

--

Paul Loy
paul@keteracel.com
http://www.keteracel.com/paul

3

Wouldn't a "refresh" of the index right before the search solve this?
(http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/
indices/refresh/)

On Jun 3, 9:45 pm, Berkay Mollamustafaoglu mber...@gmail.com wrote:

Hi Paul,

It sounds like you have a similar requirement to us. After a brief
discussion with Shay, I've created the following issue on githubhttp://github.com/elasticsearch/elasticsearch/issues#issue/204

It is not clear what the best way to handle this requirement is. Having get
method work in real-time may be one potential solution. There may be better
ways to solve this as well.

Regards,
Berkay Mollamustafaogluhttp://www.ifountain.com
mberkay on yahoo, google and skype

On Thu, Jun 3, 2010 at 12:11 PM, Paul Loy ketera...@gmail.com wrote:

Hi Shay,

the current model we have for our data allows partial Documents to be
indexed. At some point in the future we may then add the missing fields.
Currently I do this like so (in pseudo):

function update(PartialDoc update) {
PartialDoc doc = get();
doc.merge(update);
index(doc);
}

The problem is that the initial indexing of the doc may not have happened
before we call get so after the merge+index,* *we have lost the
previously indexed fields.

Is there a way to specify a consistency level in Elasticsearch (like
Cassandra), or couls Elastic Search internally provide update functionality?

Thanks,

Paul

--

Paul Loy
p...@keteracel.com
http://www.keteracel.com/paul

4

Yes, but calling refresh before (or after) ever index operation would have a
very high performance penalty, hence not really a feasible solution when
performance is a concern.

Regards,
Berkay Mollamustafaoglu
http://www.ifountain.com
mberkay on yahoo, google and skype

On Thu, Jun 3, 2010 at 3:46 PM, Tal talsalmona@gmail.com wrote:

Wouldn't a "refresh" of the index right before the search solve this?
(http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/
indices/refresh/)

On Jun 3, 9:45 pm, Berkay Mollamustafaoglu mber...@gmail.com wrote:

Hi Paul,

It sounds like you have a similar requirement to us. After a brief
discussion with Shay, I've created the following issue on githubhttp://
Issues · elastic/elasticsearch · GitHub

It is not clear what the best way to handle this requirement is. Having
get
method work in real-time may be one potential solution. There may be
better
ways to solve this as well.

Regards,
Berkay Mollamustafaogluhttp://www.ifountain.com
mberkay on yahoo, google and skype

On Thu, Jun 3, 2010 at 12:11 PM, Paul Loy ketera...@gmail.com wrote:

Hi Shay,

the current model we have for our data allows partial Documents to be
indexed. At some point in the future we may then add the missing
fields.
Currently I do this like so (in pseudo):

function update(PartialDoc update) {
PartialDoc doc = get();
doc.merge(update);
index(doc);
}

The problem is that the initial indexing of the doc may not have
happened
before we call get so after the merge+index,* *we have lost the
previously indexed fields.

Is there a way to specify a consistency level in Elasticsearch (like
Cassandra), or couls Elastic Search internally provide update
functionality?

Thanks,

Paul

--

Paul Loy
p...@keteracel.com
http://www.keteracel.com/paul

5

The problem with refresh and get is that I added a refresh for a demo but it
can still break. Also, as Berkay has stated, it would be very costly to
refresh an entire index in order to retrieve a single record consistently.

Also, what happens if another index request comes in after the refresh but
before the get?

I think it would be much better to handle updates as a special case inside
Elastic Search. So this is a slightly different problem/solution than issue
204 in my opinion.

Thanks,

Paul.

On Thu, Jun 3, 2010 at 8:48 PM, Berkay Mollamustafaoglu
mberkay@gmail.comwrote:

Yes, but calling refresh before (or after) ever index operation would have
a very high performance penalty, hence not really a feasible solution when
performance is a concern.

Regards,
Berkay Mollamustafaoglu

http://www.ifountain.com
mberkay on yahoo, google and skype

On Thu, Jun 3, 2010 at 3:46 PM, Tal talsalmona@gmail.com wrote:

Wouldn't a "refresh" of the index right before the search solve this?
(http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/
indices/refresh/http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/indices/refresh/
)

On Jun 3, 9:45 pm, Berkay Mollamustafaoglu mber...@gmail.com wrote:

Hi Paul,

It sounds like you have a similar requirement to us. After a brief
discussion with Shay, I've created the following issue on githubhttp://
Issues · elastic/elasticsearch · GitHub

It is not clear what the best way to handle this requirement is. Having
get
method work in real-time may be one potential solution. There may be
better
ways to solve this as well.

Regards,
Berkay Mollamustafaogluhttp://www.ifountain.com
mberkay on yahoo, google and skype

On Thu, Jun 3, 2010 at 12:11 PM, Paul Loy ketera...@gmail.com wrote:

Hi Shay,

the current model we have for our data allows partial Documents to be
indexed. At some point in the future we may then add the missing
fields.
Currently I do this like so (in pseudo):

function update(PartialDoc update) {
PartialDoc doc = get();
doc.merge(update);
index(doc);
}

The problem is that the initial indexing of the doc may not have
happened
before we call get so after the merge+index,* *we have lost the
previously indexed fields.

Is there a way to specify a consistency level in Elasticsearch (like
Cassandra), or couls Elastic Search internally provide update
functionality?

Thanks,

Paul

--

Paul Loy
p...@keteracel.com
http://www.keteracel.com/paul

--

Paul Loy
paul@keteracel.com
http://www.keteracel.com/paul

6

Yes, it is. What you want is something similar to optimistic locking (aka
etag and "If-Match" in REST). When you do a GET or search, you get back
etags with each resource, and when you do an index, you provide it, and if
they don't match, then it fails.

All are possible, real time gets, etags, and so on. But, I am currently
focusing on making elasticsearch the best search solution out there. There
are some search features that I am focusing for the next release.

-shay.banon

On Fri, Jun 4, 2010 at 1:59 AM, Paul Loy keteracel@gmail.com wrote:

The problem with refresh and get is that I added a refresh for a demo but
it can still break. Also, as Berkay has stated, it would be very costly to
refresh an entire index in order to retrieve a single record consistently.

Also, what happens if another index request comes in after the refresh but
before the get?

I think it would be much better to handle updates as a special case inside
Elastic Search. So this is a slightly different problem/solution than issue
204 in my opinion.

Thanks,

Paul.

On Thu, Jun 3, 2010 at 8:48 PM, Berkay Mollamustafaoglu <mberkay@gmail.com

wrote:

Yes, but calling refresh before (or after) ever index operation would have
a very high performance penalty, hence not really a feasible solution when
performance is a concern.

Regards,
Berkay Mollamustafaoglu

http://www.ifountain.com
mberkay on yahoo, google and skype

On Thu, Jun 3, 2010 at 3:46 PM, Tal talsalmona@gmail.com wrote:

Wouldn't a "refresh" of the index right before the search solve this?
(http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/
indices/refresh/http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/indices/refresh/
)

On Jun 3, 9:45 pm, Berkay Mollamustafaoglu mber...@gmail.com wrote:

Hi Paul,

It sounds like you have a similar requirement to us. After a brief
discussion with Shay, I've created the following issue on githubhttp://
Issues · elastic/elasticsearch · GitHub

It is not clear what the best way to handle this requirement is. Having
get
method work in real-time may be one potential solution. There may be
better
ways to solve this as well.

Regards,
Berkay Mollamustafaogluhttp://www.ifountain.com
mberkay on yahoo, google and skype

On Thu, Jun 3, 2010 at 12:11 PM, Paul Loy ketera...@gmail.com wrote:

Hi Shay,

the current model we have for our data allows partial Documents to be
indexed. At some point in the future we may then add the missing
fields.
Currently I do this like so (in pseudo):

function update(PartialDoc update) {
PartialDoc doc = get();
doc.merge(update);
index(doc);
}

The problem is that the initial indexing of the doc may not have
happened
before we call get so after the merge+index,* *we have lost the
previously indexed fields.

Is there a way to specify a consistency level in Elasticsearch (like
Cassandra), or couls Elastic Search internally provide update
functionality?

Thanks,

Paul

--

Paul Loy
p...@keteracel.com
http://www.keteracel.com/paul

--

Paul Loy
paul@keteracel.com

http://www.keteracel.com/paul