Can't combine "order" with "size" in 0.90.0?

Elastic Stack Elasticsearch
1

I have a simple index with 4 docs containing a "tag" field, with the values
"alpha", "beta", "delta" and "gamma", respectively. It appears that when I
restrict the "size" to less than the total number of hits, the "order" is
ignored. The same request works as expected in 0.20.6.

request:
{
"from" : 0,
"size" : 2,
"query" : {
"match_all" : { }
},
"version" : true,
"sort" : [ {
"tag" : {
"order" : "asc"
}
} ]
}

response:
{
"took" : 17,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : null,
"hits" : [ {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "bruovacpmo",
"_version" : 1,
"_score" : null, "_source" : {"tag":"alpha"},
"sort" : [ "alpha" ]
}, {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "29bv3m8n8a",
"_version" : 1,
"_score" : null, "_source" : {"tag":"gamma"},
"sort" : [ "gamma" ]
} ]
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hey,

can you provide a complete gist to reproduce (including mapping)? I just
tried this (with 0.90.0)

curl -v -X PUT localhost:9200/index/type/1 -d '{ "tags" : [ "alpha" ] }'
curl -v -X PUT localhost:9200/index/type/2 -d '{ "tags" : [ "beta" ] }'
curl -v -X PUT localhost:9200/index/type/3 -d '{ "tags" : [ "gamma" ] }'
curl -v -X PUT localhost:9200/index/type/4 -d '{ "tags" : [ "delta" ] }'

curl -X POST localhost:9200/index/type/_search -d '
{
"from" : 0,
"size" : 3,
"query" : {
"match_all" : { }
},
"version" : true,
"sort" : [ {
"tags" : {
"order" : "asc"
}
} ]
}
'

The query returns "alpha" "beta" "delta" hits and looks ok. Maybe I
overlooked something from your query.

--Alex

On Fri, May 3, 2013 at 4:59 AM, Eric Jain eric.jain@gmail.com wrote:

I have a simple index with 4 docs containing a "tag" field, with the
values "alpha", "beta", "delta" and "gamma", respectively. It appears that
when I restrict the "size" to less than the total number of hits, the
"order" is ignored. The same request works as expected in 0.20.6.

request:
{
"from" : 0,
"size" : 2,
"query" : {
"match_all" : { }
},
"version" : true,
"sort" : [ {
"tag" : {
"order" : "asc"
}
} ]
}

response:
{
"took" : 17,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : null,
"hits" : [ {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "bruovacpmo",
"_version" : 1,
"_score" : null, "_source" : {"tag":"alpha"},
"sort" : [ "alpha" ]
}, {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "29bv3m8n8a",
"_version" : 1,
"_score" : null, "_source" : {"tag":"gamma"},
"sort" : [ "gamma" ]
} ]
}
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

To replicate the issue, create an index with a single shard and no replicas.

On Fri, May 3, 2013 at 2:41 AM, Alexander Reelsen alr@spinscale.de wrote:

Hey,

can you provide a complete gist to reproduce (including mapping)? I just
tried this (with 0.90.0)

curl -v -X PUT localhost:9200/index/type/1 -d '{ "tags" : [ "alpha" ] }'
curl -v -X PUT localhost:9200/index/type/2 -d '{ "tags" : [ "beta" ] }'
curl -v -X PUT localhost:9200/index/type/3 -d '{ "tags" : [ "gamma" ] }'
curl -v -X PUT localhost:9200/index/type/4 -d '{ "tags" : [ "delta" ] }'

curl -X POST localhost:9200/index/type/_search -d '
{
"from" : 0,
"size" : 3,
"query" : {
"match_all" : { }
},
"version" : true,
"sort" : [ {
"tags" : {
"order" : "asc"
}
} ]
}
'

The query returns "alpha" "beta" "delta" hits and looks ok. Maybe I
overlooked something from your query.

--Alex

On Fri, May 3, 2013 at 4:59 AM, Eric Jain eric.jain@gmail.com wrote:

I have a simple index with 4 docs containing a "tag" field, with the
values "alpha", "beta", "delta" and "gamma", respectively. It appears that
when I restrict the "size" to less than the total number of hits, the
"order" is ignored. The same request works as expected in 0.20.6.

request:
{
"from" : 0,
"size" : 2,
"query" : {
"match_all" : { }
},
"version" : true,
"sort" : [ {
"tag" : {
"order" : "asc"
}
} ]
}

response:
{
"took" : 17,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : null,
"hits" : [ {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "bruovacpmo",
"_version" : 1,
"_score" : null, "_source" : {"tag":"alpha"},
"sort" : [ "alpha" ]
}, {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "29bv3m8n8a",
"_version" : 1,
"_score" : null, "_source" : {"tag":"gamma"},
"sort" : [ "gamma" ]
} ]
}
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/W5s1KypYcYw/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

That one is tricky... I can reproduce this and that is in-fact a bug. I
opened an issue with some explanation.

thank you very much for raising this.

simon
On Friday, May 3, 2013 8:21:07 PM UTC+2, Eric Jain wrote:

To replicate the issue, create an index with a single shard and no
replicas.

On Fri, May 3, 2013 at 2:41 AM, Alexander Reelsen <a...@spinscale.de<javascript:>>
wrote:

Hey,

can you provide a complete gist to reproduce (including mapping)? I just
tried this (with 0.90.0)

curl -v -X PUT localhost:9200/index/type/1 -d '{ "tags" : [ "alpha" ]
}'
curl -v -X PUT localhost:9200/index/type/2 -d '{ "tags" : [ "beta" ] }'
curl -v -X PUT localhost:9200/index/type/3 -d '{ "tags" : [ "gamma" ]
}'
curl -v -X PUT localhost:9200/index/type/4 -d '{ "tags" : [ "delta" ]
}'

curl -X POST localhost:9200/index/type/_search -d '
{
"from" : 0,
"size" : 3,
"query" : {
"match_all" : { }
},
"version" : true,
"sort" : [ {
"tags" : {
"order" : "asc"
}
} ]
}
'

The query returns "alpha" "beta" "delta" hits and looks ok. Maybe I
overlooked something from your query.

--Alex

On Fri, May 3, 2013 at 4:59 AM, Eric Jain <eric...@gmail.com<javascript:>>
wrote:

I have a simple index with 4 docs containing a "tag" field, with the
values "alpha", "beta", "delta" and "gamma", respectively. It appears
that
when I restrict the "size" to less than the total number of hits, the
"order" is ignored. The same request works as expected in 0.20.6.

request:
{
"from" : 0,
"size" : 2,
"query" : {
"match_all" : { }
},
"version" : true,
"sort" : [ {
"tag" : {
"order" : "asc"
}
} ]
}

response:
{
"took" : 17,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : null,
"hits" : [ {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "bruovacpmo",
"_version" : 1,
"_score" : null, "_source" : {"tag":"alpha"},
"sort" : [ "alpha" ]
}, {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "29bv3m8n8a",
"_version" : 1,
"_score" : null, "_source" : {"tag":"gamma"},
"sort" : [ "gamma" ]
} ]
}
}

--
You received this message because you are subscribed to the Google
Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to elasticsearc...@googlegroups.com <javascript:>.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/elasticsearch/W5s1KypYcYw/unsubscribe?hl=en-US.

To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

@Alex, I guess you added the docs to quickly so they didn't end up in
different segments. for this to fail you need to have at least two segments
:slight_smile:

simon

On Sunday, May 5, 2013 12:46:06 AM UTC+2, simonw wrote:

That one is tricky... I can reproduce this and that is in-fact a bug. I
opened an issue with some explanation.

BytesRefOrdValComparator ignores highest value in a segment during binarySearch · Issue #2991 · elastic/elasticsearch · GitHub

thank you very much for raising this.

simon
On Friday, May 3, 2013 8:21:07 PM UTC+2, Eric Jain wrote:

To replicate the issue, create an index with a single shard and no
replicas.

On Fri, May 3, 2013 at 2:41 AM, Alexander Reelsen a...@spinscale.de
wrote:

Hey,

can you provide a complete gist to reproduce (including mapping)? I
just
tried this (with 0.90.0)

curl -v -X PUT localhost:9200/index/type/1 -d '{ "tags" : [ "alpha" ]
}'
curl -v -X PUT localhost:9200/index/type/2 -d '{ "tags" : [ "beta" ]
}'
curl -v -X PUT localhost:9200/index/type/3 -d '{ "tags" : [ "gamma" ]
}'
curl -v -X PUT localhost:9200/index/type/4 -d '{ "tags" : [ "delta" ]
}'

curl -X POST localhost:9200/index/type/_search -d '
{
"from" : 0,
"size" : 3,
"query" : {
"match_all" : { }
},
"version" : true,
"sort" : [ {
"tags" : {
"order" : "asc"
}
} ]
}
'

The query returns "alpha" "beta" "delta" hits and looks ok. Maybe I
overlooked something from your query.

--Alex

On Fri, May 3, 2013 at 4:59 AM, Eric Jain eric...@gmail.com wrote:

I have a simple index with 4 docs containing a "tag" field, with the
values "alpha", "beta", "delta" and "gamma", respectively. It appears
that
when I restrict the "size" to less than the total number of hits, the
"order" is ignored. The same request works as expected in 0.20.6.

request:
{
"from" : 0,
"size" : 2,
"query" : {
"match_all" : { }
},
"version" : true,
"sort" : [ {
"tag" : {
"order" : "asc"
}
} ]
}

response:
{
"took" : 17,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : null,
"hits" : [ {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "bruovacpmo",
"_version" : 1,
"_score" : null, "_source" : {"tag":"alpha"},
"sort" : [ "alpha" ]
}, {
"_index" : "c1cp147rfj",
"_type" : "event",
"_id" : "29bv3m8n8a",
"_version" : 1,
"_score" : null, "_source" : {"tag":"gamma"},
"sort" : [ "gamma" ]
} ]
}
}

--
You received this message because you are subscribed to the Google
Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/elasticsearch/W5s1KypYcYw/unsubscribe?hl=en-US.

To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

btw I'm still seeing an ordering issue in 0.90.1: When retrieving 10 docs
(out of 20) ordered by the value of a field, one of the first 9 docs
sometimes appears in position 10. Haven't been able to reproduce this with
a standalone elasticsearch, only in embedded mode.

Anyone else notice this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

hey there are several situations where this can happen. ie if another doc
has the same value we use the internal doc ID as a tie-breaker. So you you
page to the second page and hit a different shard the docs might have a
different internal Id due to segment merges etc. so the order is slightly
different. you can use a preference on the search request so make sure you
hit the same shards. is it possible that you see this?

simon

On Thursday, June 20, 2013 12:20:12 AM UTC+2, Eric Jain wrote:

btw I'm still seeing an ordering issue in 0.90.1: When retrieving 10 docs
(out of 20) ordered by the value of a field, one of the first 9 docs
sometimes appears in position 10. Haven't been able to reproduce this with
a standalone elasticsearch, only in embedded mode.

Anyone else notice this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Hi Simon,

do I read it correctly that using preference is now de facto recommended
for pagination with order?

Regards,
Lukas

On Thu, Jun 20, 2013 at 9:55 AM, simonw
simon.willnauer@elasticsearch.comwrote:

hey there are several situations where this can happen. ie if another doc
has the same value we use the internal doc ID as a tie-breaker. So you you
page to the second page and hit a different shard the docs might have a
different internal Id due to segment merges etc. so the order is slightly
different. you can use a preference on the search request so make sure you
hit the same shards. is it possible that you see this?

simon

On Thursday, June 20, 2013 12:20:12 AM UTC+2, Eric Jain wrote:

btw I'm still seeing an ordering issue in 0.90.1: When retrieving 10 docs
(out of 20) ordered by the value of a field, one of the first 9 docs
sometimes appears in position 10. Haven't been able to reproduce this with
a standalone elasticsearch, only in embedded mode.

Anyone else notice this?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Lukas,

what do you mean by de facto recommended? I mean you have realtime changes
so you can use preference to reduce the risk but you might hit a reopen and
this will still make you list different. Yet, if you use scroll you can be
on the safe side but that will hold on to resources longer.

simon

On Thursday, June 20, 2013 10:14:39 AM UTC+2, Lukáš Vlček wrote:

Hi Simon,

do I read it correctly that using preference is now de facto recommended
for pagination with order?

Regards,
Lukas

On Thu, Jun 20, 2013 at 9:55 AM, simonw <simon.w...@elasticsearch.com<javascript:>

wrote:

hey there are several situations where this can happen. ie if another doc
has the same value we use the internal doc ID as a tie-breaker. So you you
page to the second page and hit a different shard the docs might have a
different internal Id due to segment merges etc. so the order is slightly
different. you can use a preference on the search request so make sure you
hit the same shards. is it possible that you see this?

simon

On Thursday, June 20, 2013 12:20:12 AM UTC+2, Eric Jain wrote:

btw I'm still seeing an ordering issue in 0.90.1: When retrieving 10
docs (out of 20) ordered by the value of a field, one of the first 9 docs
sometimes appears in position 10. Haven't been able to reproduce this with
a standalone elasticsearch, only in embedded mode.

Anyone else notice this?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

On Thu, Jun 20, 2013 at 12:55 AM, simonw
simon.willnauer@elasticsearch.com wrote:

is it possible that you see this?

No, because the field I am sorting on has a unique value for each document.

Strangely, I'm unable to reproduce the issue today, even though the
test case is quite simple. I'll try again tomorrow, could be an
odd-day-only bug :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

Hi Simon,

I was looking at the fix to better understand what was going on. I think I
am basically getting it (the -1 was the problem as you explained in the
#2991).

Now, if I understand correctly, one might still get different results (i.e.
two same queries yielding different order of hits) when matching documents
have same value and they can not all fit into the size window. When docs
have the same value (by which they are being sorted on) then the sort is
influenced by internal docID and this internal docID changes when these
documents are reindexed and seen after reopen. Is that what is happening?

Thanks,
Lukas

On Thu, Jun 20, 2013 at 4:29 PM, simonw
simon.willnauer@elasticsearch.comwrote:

Lukas,

what do you mean by de facto recommended? I mean you have realtime changes
so you can use preference to reduce the risk but you might hit a reopen and
this will still make you list different. Yet, if you use scroll you can be
on the safe side but that will hold on to resources longer.

simon

On Thursday, June 20, 2013 10:14:39 AM UTC+2, Lukáš Vlček wrote:

Hi Simon,

do I read it correctly that using preference is now de facto recommended
for pagination with order?

Regards,
Lukas

On Thu, Jun 20, 2013 at 9:55 AM, simonw <simon.w...@**elasticsearch.com>wrote:

hey there are several situations where this can happen. ie if another
doc has the same value we use the internal doc ID as a tie-breaker. So you
you page to the second page and hit a different shard the docs might have a
different internal Id due to segment merges etc. so the order is slightly
different. you can use a preference on the search request so make sure you
hit the same shards. is it possible that you see this?

simon

On Thursday, June 20, 2013 12:20:12 AM UTC+2, Eric Jain wrote:

btw I'm still seeing an ordering issue in 0.90.1: When retrieving 10
docs (out of 20) ordered by the value of a field, one of the first 9 docs
sometimes appears in position 10. Haven't been able to reproduce this with
a standalone elasticsearch, only in embedded mode.

Anyone else notice this?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

On Wednesday, June 19, 2013 3:20:12 PM UTC-7, Eric Jain wrote:

btw I'm still seeing an ordering issue in 0.90.1: When retrieving 10 docs
(out of 20) ordered by the value of a field, one of the first 9 docs
sometimes appears in position 10. Haven't been able to reproduce this with
a standalone elasticsearch, only in embedded mode.

Turns out the field I was sorting on was not indexed, so I guess
elasticsearch decided to return the results in doc ID order rather than
failing...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.