Sorting date fields

Elastic Stack Elasticsearch
1

Hi all,

I have a question on how sorting during queries works in elasticsearch.

I have an index with a custom date format field, on which the sort is applied.
When quering the index for a given keywork, results are provided with the given
sort.

However, I've observed that some documents are not present in the result set. I
would have expected these results to be part of the result set as it would be
in relational systems using the SQL ORDER BY statement. I've verified that
these missing documents are covered by the query using the explain api.
According to the documentation, score computation ist not performed when using
sorts on fields.

Maybe someone can provide more information on how sorting is done?

I am using Elasticsearch 1.0.0RC1 on debian whezzy with openjdk7-jdk.

Thanks, Adrian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140225212301.GA22436%40server1.850nm.net.
For more options, visit https://groups.google.com/groups/opt_out.

2

ES loads the values of the fields to sort on into memory cache.

You should update to 1.0.0, maybe you hit a bug that has been fixed.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHo_N3LXo6NiGztL5%3D1GaqVG4QOTCX32OuAjYeOhGfFng%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

On Tue, Feb 25, 2014 at 11:11:13PM +0100, joergprante@gmail.com wrote:

Jörg,

ES loads the values of the fields to sort on into memory cache.

Yes, I've read that - is it known when these caches are flushed?

You should update to 1.0.0, maybe you hit a bug that has been fixed.

I'll do that. I am just wondering if I am missing something ..

Best regards, Adrian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140225222829.GB22436%40server1.850nm.net.
For more options, visit https://groups.google.com/groups/opt_out.

4

For the cache, see

http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/index-modules-fielddata.html

By default, the field cache size is unbounded, and does not expire. For
sort, it means that each field to sort is examined, all values of the field
are loaded, so the in-memory sorting can take place. It's exactly the same
what Lucene is executing.

With the default settings of the field cache, sort is working alright
(unless the field values will exceed the available memory)

Maybe you can set up an example of your sort as a demo, so that the error
can be reproduced?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF4zOnB5TJddf3EZTgGtDqXSBzJAHpZAPo8biUE35wGQg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

On Wed, Feb 26, 2014 at 12:21:26AM +0100, joergprante@gmail.com wrote:

Jörg,

sorry for the late answer.

Maybe you can set up an example of your sort as a demo, so that the error
can be reproduced?

It turned out that this behaviour was caused by me, since documents contained
the wrong timestamp on the sorted field. After fixing this, the results were as
expected.

Thanks for the pointers and your help,

Adrian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140304221819.GA31519%40server1.850nm.net.
For more options, visit https://groups.google.com/groups/opt_out.