External versioning and performance

Hi there,

Reading this: http://www.elasticsearch.org/guide/reference/api/index_/

We might want to start using the epoch as the version for the stuff we
index - are there any drawbacks for using such large integers as version
numbers?

Better safe than sorry...

Thanks,

Itamar

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

there might be a very small impact on the index size. other than that no!
This is totally practical and I used it in production before.

simon

On Sunday, May 19, 2013 7:19:59 PM UTC+2, Itamar Syn-Hershko wrote:

Hi there,

Reading this: Elasticsearch Platform — Find real-time answers at scale | Elastic

We might want to start using the epoch as the version for the stuff we
index - are there any drawbacks for using such large integers as version
numbers?

Better safe than sorry...

Thanks,

Itamar

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

That's great to hear! thanks

On Mon, May 20, 2013 at 12:40 AM, simonw
simon.willnauer@elasticsearch.comwrote:

there might be a very small impact on the index size. other than that no!
This is totally practical and I used it in production before.

simon

On Sunday, May 19, 2013 7:19:59 PM UTC+2, Itamar Syn-Hershko wrote:

Hi there,

Reading this: http://www.Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/index_/http://www.elasticsearch.org/guide/reference/api/index_/

We might want to start using the epoch as the version for the stuff we
index - are there any drawbacks for using such large integers as version
numbers?

Better safe than sorry...

Thanks,

Itamar

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Simon, is there a way to know what was the previous version number of a
document when indexing using external version type (without querying for
it)?

We need a way to know whether the indexing operation overrode an existing
document or not. Currently we do this by simply checking if
indexResponse.getVersion == 1 (meaning, this was a fresh insert). Maybe the
indexResponse when using external versioning should return the previous
version number, or 0 if it didn't exist?

I'm using 0.90

On Mon, May 20, 2013 at 12:40 AM, simonw
simon.willnauer@elasticsearch.comwrote:

there might be a very small impact on the index size. other than that no!
This is totally practical and I used it in production before.

simon

On Sunday, May 19, 2013 7:19:59 PM UTC+2, Itamar Syn-Hershko wrote:

Hi there,

Reading this: http://www.Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/index_/http://www.elasticsearch.org/guide/reference/api/index_/

We might want to start using the epoch as the version for the stuff we
index - are there any drawbacks for using such large integers as version
numbers?

Better safe than sorry...

Thanks,

Itamar

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I don't think there is a way to do this at this point. The cheapest way
seems to be using HEAD

curl -XHEAD 'http://localhost:9200/twitter/tweet/1'

this has only http overhead, the version needs to be loaded anyway if you
index.

simon

On Monday, May 20, 2013 3:10:27 PM UTC+2, Itamar Syn-Hershko wrote:

Simon, is there a way to know what was the previous version number of a
document when indexing using external version type (without querying for
it)?

We need a way to know whether the indexing operation overrode an existing
document or not. Currently we do this by simply checking if
indexResponse.getVersion == 1 (meaning, this was a fresh insert). Maybe the
indexResponse when using external versioning should return the previous
version number, or 0 if it didn't exist?

I'm using 0.90

On Mon, May 20, 2013 at 12:40 AM, simonw <simon.w...@elasticsearch.com<javascript:>

wrote:

there might be a very small impact on the index size. other than that no!
This is totally practical and I used it in production before.

simon

On Sunday, May 19, 2013 7:19:59 PM UTC+2, Itamar Syn-Hershko wrote:

Hi there,

Reading this: http://www.Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/index_/http://www.elasticsearch.org/guide/reference/api/index_/

We might want to start using the epoch as the version for the stuff we
index - are there any drawbacks for using such large integers as version
numbers?

Better safe than sorry...

Thanks,

Itamar

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

This is one extra HTTP request I would like to avoid - we are going to do
this for every one of the many docs we push hourly

What I was thinking is this:

When indexing, you get back in the response the version number that was
assigned to the document you put. This makes sense, and quite useful in
many scenarios - for example in ours, where we check to see if it == 1 and
if it does we know its a new entry and can do perform some extra actions.

When specifying a version yourself by using the external version type, you
already know what version to expect in the response if the indexing
operation succeeds. Hence, the version returned is quite meaningless. In
this scenario, I would really like to get the PREVIOUS version. For us this
would mean knowing whether this is a new entry or not.

As far as I'm concerned, even an indication if this was an update or an
insert is just fine.

I'll be happy to provide a pull request with this change if this helps :slight_smile:

On Mon, May 20, 2013 at 9:50 PM, simonw
simon.willnauer@elasticsearch.comwrote:

I don't think there is a way to do this at this point. The cheapest way
seems to be using HEAD

curl -XHEAD 'http://localhost:9200/twitter/tweet/1'

this has only http overhead, the version needs to be loaded anyway if you
index.

simon

On Monday, May 20, 2013 3:10:27 PM UTC+2, Itamar Syn-Hershko wrote:

Simon, is there a way to know what was the previous version number of a
document when indexing using external version type (without querying for
it)?

We need a way to know whether the indexing operation overrode an existing
document or not. Currently we do this by simply checking if
indexResponse.getVersion == 1 (meaning, this was a fresh insert). Maybe the
indexResponse when using external versioning should return the previous
version number, or 0 if it didn't exist?

I'm using 0.90

On Mon, May 20, 2013 at 12:40 AM, simonw <simon.w...@**elasticsearch.com>wrote:

there might be a very small impact on the index size. other than that
no! This is totally practical and I used it in production before.

simon

On Sunday, May 19, 2013 7:19:59 PM UTC+2, Itamar Syn-Hershko wrote:

Hi there,

Reading this: http://www.elasticsearch.org/guide/reference/api/
index_/ http://www.elasticsearch.org/guide/reference/api/index_/

We might want to start using the epoch as the version for the stuff we
index - are there any drawbacks for using such large integers as version
numbers?

Better safe than sorry...

Thanks,

Itamar

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Monday, May 20, 2013 11:40:16 PM UTC+2, Itamar Syn-Hershko wrote:

This is one extra HTTP request I would like to avoid - we are going to do
this for every one of the many docs we push hourly

I know :slight_smile: I agree as well... why don't you open an issue for this?

simon

What I was thinking is this:

When indexing, you get back in the response the version number that was
assigned to the document you put. This makes sense, and quite useful in
many scenarios - for example in ours, where we check to see if it == 1 and
if it does we know its a new entry and can do perform some extra actions.

When specifying a version yourself by using the external version type, you
already know what version to expect in the response if the indexing
operation succeeds. Hence, the version returned is quite meaningless. In
this scenario, I would really like to get the PREVIOUS version. For us this
would mean knowing whether this is a new entry or not.

As far as I'm concerned, even an indication if this was an update or an
insert is just fine.

I'll be happy to provide a pull request with this change if this helps :slight_smile:

On Mon, May 20, 2013 at 9:50 PM, simonw <simon.w...@elasticsearch.com<javascript:>

wrote:

I don't think there is a way to do this at this point. The cheapest way
seems to be using HEAD

curl -XHEAD 'http://localhost:9200/twitter/tweet/1'

this has only http overhead, the version needs to be loaded anyway if you
index.

simon

On Monday, May 20, 2013 3:10:27 PM UTC+2, Itamar Syn-Hershko wrote:

Simon, is there a way to know what was the previous version number of a
document when indexing using external version type (without querying for
it)?

We need a way to know whether the indexing operation overrode an
existing document or not. Currently we do this by simply checking if
indexResponse.getVersion == 1 (meaning, this was a fresh insert). Maybe the
indexResponse when using external versioning should return the previous
version number, or 0 if it didn't exist?

I'm using 0.90

On Mon, May 20, 2013 at 12:40 AM, simonw <simon.w...@**elasticsearch.com

wrote:

there might be a very small impact on the index size. other than that
no! This is totally practical and I used it in production before.

simon

On Sunday, May 19, 2013 7:19:59 PM UTC+2, Itamar Syn-Hershko wrote:

Hi there,

Reading this: http://www.elasticsearch.org/guide/*reference/api/
*index_/ http://www.elasticsearch.org/guide/reference/api/index_/

We might want to start using the epoch as the version for the stuff we
index - are there any drawbacks for using such large integers as version
numbers?

Better safe than sorry...

Thanks,

Itamar

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Tue, May 21, 2013 at 1:02 PM, simonw
simon.willnauer@elasticsearch.comwrote:

On Monday, May 20, 2013 11:40:16 PM UTC+2, Itamar Syn-Hershko wrote:

This is one extra HTTP request I would like to avoid - we are going to do
this for every one of the many docs we push hourly

I know :slight_smile: I agree as well... why don't you open an issue for this?

simon

What I was thinking is this:

When indexing, you get back in the response the version number that was
assigned to the document you put. This makes sense, and quite useful in
many scenarios - for example in ours, where we check to see if it == 1 and
if it does we know its a new entry and can do perform some extra actions.

When specifying a version yourself by using the external version type,
you already know what version to expect in the response if the indexing
operation succeeds. Hence, the version returned is quite meaningless. In
this scenario, I would really like to get the PREVIOUS version. For us this
would mean knowing whether this is a new entry or not.

As far as I'm concerned, even an indication if this was an update or an
insert is just fine.

I'll be happy to provide a pull request with this change if this helps :slight_smile:

On Mon, May 20, 2013 at 9:50 PM, simonw <simon.w...@**elasticsearch.com>wrote:

I don't think there is a way to do this at this point. The cheapest way
seems to be using HEAD

curl -XHEAD 'http://localhost:9200/**twitter/tweet/1'

this has only http overhead, the version needs to be loaded anyway if
you index.

simon

On Monday, May 20, 2013 3:10:27 PM UTC+2, Itamar Syn-Hershko wrote:

Simon, is there a way to know what was the previous version number of a
document when indexing using external version type (without querying for
it)?

We need a way to know whether the indexing operation overrode an
existing document or not. Currently we do this by simply checking if
indexResponse.getVersion == 1 (meaning, this was a fresh insert). Maybe the
indexResponse when using external versioning should return the previous
version number, or 0 if it didn't exist?

I'm using 0.90

On Mon, May 20, 2013 at 12:40 AM, simonw <simon.w...@**
elasticsearch.com> wrote:

there might be a very small impact on the index size. other than that
no! This is totally practical and I used it in production before.

simon

On Sunday, May 19, 2013 7:19:59 PM UTC+2, Itamar Syn-Hershko wrote:

Hi there,

Reading this: http://www.elasticsearch**.org/guide/**
reference/api/index_/http://www.elasticsearch.org/guide/reference/api/index_/

We might want to start using the epoch as the version for the stuff
we index - are there any drawbacks for using such large integers as version
numbers?

Better safe than sorry...

Thanks,

Itamar

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.**com.

For more options, visit https://groups.google.com/**grou**ps/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.