ES 1.2.1 sort by _timestamp

Elastic Stack Elasticsearch
1

Hey all,

I have a question regarding sorting by _timestamp.

The _timestamp "enabled" setting is set to true, everything else is not
explicitly specified (so I assume default values are used).
I have an application inserting new documents periodically and I would
assume that _timestamp gets filled with the current timestamp.

After a while I perform the following sort
curl -XPOST 'http://localhost:9200/my_index/my_type/_search?pretty' -d ' {
"sort" : [
{ "_timestamp" : {"order" : "desc" } }
]
}

I would expect to always get the last 10 inserted documents, which is not
the case.
Sometimes I get the most recent ones, sometimes I get some historical data.

Thanks for your help.

BR,
Stefan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1d81d216-f393-4618-afb7-74d71aa7841c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Do you set timestamp value from you client or do you let ES fill them for
you?

Do you run more than one node? Are the clocks on your nodes running
synchronously?

Jörg

On Thu, Jun 12, 2014 at 2:13 PM, Stefan Eberl cpppwner@gmail.com wrote:

Hey all,

I have a question regarding sorting by _timestamp.

The _timestamp "enabled" setting is set to true, everything else is not
explicitly specified (so I assume default values are used).
I have an application inserting new documents periodically and I would
assume that _timestamp gets filled with the current timestamp.

After a while I perform the following sort
curl -XPOST 'http://localhost:9200/my_index/my_type/_search?pretty' -d ' {
"sort" : [
{ "_timestamp" : {"order" : "desc" } }
]
}

I would expect to always get the last 10 inserted documents, which is not
the case.
Sometimes I get the most recent ones, sometimes I get some historical data.

Thanks for your help.

BR,
Stefan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d81d216-f393-4618-afb7-74d71aa7841c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d81d216-f393-4618-afb7-74d71aa7841c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFVuicgKrdrAXKWsT3FgQFccXtXOTXuJi9-q3-bWJJpYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Do you set timestamp value from you client or do you let ES fill them for
you?

I let ES fill that value.

Do you run more than one node? Are the clocks on your nodes running
synchronously?

Only one node is storing data, the remaining nodes are client nodes only.
Clocks are synchronized with a time server.

BR,
Stefan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/27edfe03-f8d7-4114-95dc-129b7b6e7bfb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

This is weird. Are you sure what you are seeing is not overridden documents
(can happen if you specify the ID yourself)? Can you add the _timestamp
field to the results and verify the documents are indeed not sorted by
_timestamp?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Thu, Jun 12, 2014 at 3:13 PM, Stefan Eberl cpppwner@gmail.com wrote:

Hey all,

I have a question regarding sorting by _timestamp.

The _timestamp "enabled" setting is set to true, everything else is not
explicitly specified (so I assume default values are used).
I have an application inserting new documents periodically and I would
assume that _timestamp gets filled with the current timestamp.

After a while I perform the following sort
curl -XPOST 'http://localhost:9200/my_index/my_type/_search?pretty' -d ' {
"sort" : [
{ "_timestamp" : {"order" : "desc" } }
]
}

I would expect to always get the last 10 inserted documents, which is not
the case.
Sometimes I get the most recent ones, sometimes I get some historical data.

Thanks for your help.

BR,
Stefan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d81d216-f393-4618-afb7-74d71aa7841c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d81d216-f393-4618-afb7-74d71aa7841c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu8xcCKvC_joOdGDCKc7SMamOJh1W2jfqs4%3Dh8g%2BbysHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

On Thursday, June 12, 2014 6:52:16 PM UTC+2, Itamar Syn-Hershko wrote:

This is weird. Are you sure what you are seeing is not overridden
documents (can happen if you specify the ID yourself)? Can you add the
_timestamp field to the results and verify the documents are indeed not
sorted by _timestamp?

The id is also automatically generated by ES.
Do i need to store the _timestamp field to be able to retrieve it using
"fields" : ["_timestamp"] in my query?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c1190f3b-2707-4b1e-9151-5967a7a54733%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Possibly, because it's not provided in the _source, or just use this:

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Fri, Jun 13, 2014 at 11:26 AM, Stefan Eberl cpppwner@gmail.com wrote:

On Thursday, June 12, 2014 6:52:16 PM UTC+2, Itamar Syn-Hershko wrote:

This is weird. Are you sure what you are seeing is not overridden
documents (can happen if you specify the ID yourself)? Can you add the
_timestamp field to the results and verify the documents are indeed not
sorted by _timestamp?

The id is also automatically generated by ES.
Do i need to store the _timestamp field to be able to retrieve it using
"fields" : ["_timestamp"] in my query?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c1190f3b-2707-4b1e-9151-5967a7a54733%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c1190f3b-2707-4b1e-9151-5967a7a54733%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvRD%3DbHqX-snqSLV9p%3D%3DA3KEwhP%3DNHAiH6m%2BMz2AhRvAQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

On Friday, June 13, 2014 10:31:53 AM UTC+2, Itamar Syn-Hershko wrote:

Possibly, because it's not provided in the _source, or just use this:
Elasticsearch Platform — Find real-time answers at scale | Elastic

So your suggestion is to have my app fill an additional field, which then
gets's mapped to _timestamp, correct?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d68a20f4-8234-4bfc-b4ee-d135f948dda5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

This is just to debug this, to make sure results are indeed not sorted by
_timestamp, as you claim. Probably easier to just set _timestamp to stored.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Fri, Jun 13, 2014 at 12:49 PM, Stefan Eberl cpppwner@gmail.com wrote:

On Friday, June 13, 2014 10:31:53 AM UTC+2, Itamar Syn-Hershko wrote:

Possibly, because it's not provided in the _source, or just use this:
Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/mapping-timestamp-field.html#_path_2

So your suggestion is to have my app fill an additional field, which then
gets's mapped to _timestamp, correct?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d68a20f4-8234-4bfc-b4ee-d135f948dda5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d68a20f4-8234-4bfc-b4ee-d135f948dda5%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuuCpjAnQhGM3WnKuPd8VcgD3X_HqDcf7MA%2B%3D7dYRcjxg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.