Behavior of scrolling search with "from" parameter in newer ES versions

Gordon_Tillman · January 20, 2015, 10:33pm

Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to
newer versions. In particular, I tested the following with versions 1.2.4,
1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan), it
appears that the "from" parameter is being ignored.

example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied query,
so with from=1 no documents are returned.

curl
localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters, but
also specify a scroll parameter, we incorrectly get the single document
returned from the query.

curl
localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4&from=1&scroll=2s
| json

notes

Both of the above test cases work correctly in version 1.0.1
In newer versions (where example 2 fails), I noticed that the from
value is not present in the data that was returned from the query.

I understand with with a *search_type=scan, *this would be expected because
sorting is disabled. Also, please note this is a greatly simplified query
just for illustration purposes. I've attached a sample of an actual query
at the bottom of this post, and it does contain sorting specifications.

Is this a known issue or is this being done by design in the newer ES
versions. A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query

{
"sort": [
{
"_type": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"name_lower": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"dds_key": {
"order": "asc",
"ignore_unmapped": true
}
}
],
"query": {
"filtered": {
"filter": {
"and": [
{
"or": [
{
"term": {
"parent":
"c10ed0583104012f94e11ad0ac36f2aaf"
}
}
]
},
{
"not": {
"term": {
"vcn": "DeleteMarker"
}
}
},
{
"not": {
"exists": {
"field": "notfinalized"
}
}
}
]
},
"query": {
"match_all": {}
}
}
},
"from": 1,
"size": 1000
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mvg · January 21, 2015, 8:57am

Hi Gordon,

This from is kind of ignored for scroll search. I don't remember why that
was the case, but it seems to me that scroll search can/should take into
account the from option during the first scroll search request.

Martijn

On 20 January 2015 at 23:33, Gordon Tillman gordyt@gmail.com wrote:

Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to
newer versions. In particular, I tested the following with versions 1.2.4,
1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan), it
appears that the "from" parameter is being ignored.

example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied query,
so with from=1 no documents are returned.

curl
localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters, but
also specify a scroll parameter, we incorrectly get the single document
returned from the query.

curl
localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4&from=1&scroll=2s
| json

notes

Both of the above test cases work correctly in version 1.0.1

In newer versions (where example 2 fails), I noticed that the from
value is not present in the data that was returned from the query.

I understand with with a *search_type=scan, *this would be expected
because sorting is disabled. Also, please note this is a greatly
simplified query just for illustration purposes. I've attached a sample of
an actual query at the bottom of this post, and it does contain sorting
specifications.

Is this a known issue or is this being done by design in the newer ES
versions. A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query

{
"sort": [
{
"_type": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"name_lower": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"dds_key": {
"order": "asc",
"ignore_unmapped": true
}
}
],
"query": {
"filtered": {
"filter": {
"and": [
{
"or": [
{
"term": {
"parent":
"c10ed0583104012f94e11ad0ac36f2aaf"
}
}
]
},
{
"not": {
"term": {
"vcn": "DeleteMarker"
}
}
},
{
"not": {
"exists": {
"field": "notfinalized"
}
}
}
]
},
"query": {
"match_all": {}
}
}
},
"from": 1,
"size": 1000
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BA76Tw7_MvnXLAD_4LrQUHvxasGkEUL%2BQRBbVv_UDU3UAq1vg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Gordon_Tillman · January 21, 2015, 1:27pm

Thanks Martijn,

That is what I have observed. But is is a regression from ES version 1.0.1
and before. And I can't find anything that even implies that from is not
supported for a scroll search, except of course when search_type=scan.

I would love to get support for that back into the product.

--g

On Wednesday, January 21, 2015 at 2:57:29 AM UTC-6, Martijn v Groningen
wrote:

Hi Gordon,

This from is kind of ignored for scroll search. I don't remember why
that was the case, but it seems to me that scroll search can/should take
into account the from option during the first scroll search request.

Martijn

On 20 January 2015 at 23:33, Gordon Tillman <gor...@gmail.com
<javascript:>> wrote:

Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to
newer versions. In particular, I tested the following with versions 1.2.4,
1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan),
it appears that the "from" parameter is being ignored.

example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied query,
so with from=1 no documents are returned.

curl
localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters, but
also specify a scroll parameter, we incorrectly get the single document
returned from the query.

curl
localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4&from=1&scroll=2s
| json

notes

Both of the above test cases work correctly in version 1.0.1

In newer versions (where example 2 fails), I noticed that the
from value is not present in the data that was returned from the
query.

I understand with with a *search_type=scan, *this would be expected
because sorting is disabled. Also, please note this is a greatly
simplified query just for illustration purposes. I've attached a sample of
an actual query at the bottom of this post, and it does contain sorting
specifications.

Is this a known issue or is this being done by design in the newer ES
versions. A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query

{
"sort": [
{
"_type": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"name_lower": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"dds_key": {
"order": "asc",
"ignore_unmapped": true
}
}
],
"query": {
"filtered": {
"filter": {
"and": [
{
"or": [
{
"term": {
"parent":
"c10ed0583104012f94e11ad0ac36f2aaf"
}
}
]
},
{
"not": {
"term": {
"vcn": "DeleteMarker"
}
}
},
{
"not": {
"exists": {
"field": "notfinalized"
}
}
}
]
},
"query": {
"match_all": {}
}
}
},
"from": 1,
"size": 1000
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a4f5abd9-2f1f-4d1d-8168-260b4769d268%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mvg · January 21, 2015, 2:03pm

I agree, this should be fixed:

On 21 January 2015 at 14:27, Gordon Tillman gordyt@gmail.com wrote:

Thanks Martijn,

That is what I have observed. But is is a regression from ES version
1.0.1 and before. And I can't find anything that even implies that from
is not supported for a scroll search, except of course when
search_type=scan.

I would love to get support for that back into the product.

--g

On Wednesday, January 21, 2015 at 2:57:29 AM UTC-6, Martijn v Groningen
wrote:

Hi Gordon,

This from is kind of ignored for scroll search. I don't remember why
that was the case, but it seems to me that scroll search can/should take
into account the from option during the first scroll search request.

Martijn

On 20 January 2015 at 23:33, Gordon Tillman gor...@gmail.com wrote:

Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to
newer versions. In particular, I tested the following with versions 1.2.4,
1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan),
it appears that the "from" parameter is being ignored.

example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied
query, so with from=1 no documents are returned.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:
c10ed0583104036a94e110f0a8b5fd7d4&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters,
but also specify a scroll parameter, we incorrectly get the single document
returned from the query.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:
c10ed0583104036a94e110f0a8b5fd7d4&from=1&scroll=2s | json

notes

Both of the above test cases work correctly in version 1.0.1

In newer versions (where example 2 fails), I noticed that the
from value is not present in the data that was returned from the
query.

I understand with with a *search_type=scan, *this would be expected
because sorting is disabled. Also, please note this is a greatly
simplified query just for illustration purposes. I've attached a sample of
an actual query at the bottom of this post, and it does contain
sorting specifications.

Is this a known issue or is this being done by design in the newer ES
versions. A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query

{
"sort": [
{
"_type": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"name_lower": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"dds_key": {
"order": "asc",
"ignore_unmapped": true
}
}
],
"query": {
"filtered": {
"filter": {
"and": [
{
"or": [
{
"term": {
"parent": "
c10ed0583104012f94e11ad0ac36f2aaf"
}
}
]
},
{
"not": {
"term": {
"vcn": "DeleteMarker"
}
}
},
{
"not": {
"exists": {
"field": "notfinalized"
}
}
}
]
},
"query": {
"match_all": {}
}
}
},
"from": 1,
"size": 1000
}

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Met vriendelijke groet,

Martijn van Groningen

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BA76Tx2LjHcxMjmgo5wbFLxUYrzKR2_eHO%2B9vpCV6KGvmKZDA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Gordon_Tillman · January 21, 2015, 2:08pm

Thank you Martijn!

On Wednesday, January 21, 2015 at 8:04:23 AM UTC-6, Martijn v Groningen
wrote:

I agree, this should be fixed:
Scroll requests with `from` should throw an exception · Issue #9373 · elastic/elasticsearch · GitHub

On 21 January 2015 at 14:27, Gordon Tillman <gor...@gmail.com
<javascript:>> wrote:

Thanks Martijn,

That is what I have observed. But is is a regression from ES version
1.0.1 and before. And I can't find anything that even implies that from
is not supported for a scroll search, except of course when
search_type=scan.

I would love to get support for that back into the product.

--g

On Wednesday, January 21, 2015 at 2:57:29 AM UTC-6, Martijn v Groningen
wrote:

Hi Gordon,

This from is kind of ignored for scroll search. I don't remember why
that was the case, but it seems to me that scroll search can/should take
into account the from option during the first scroll search request.

Martijn

On 20 January 2015 at 23:33, Gordon Tillman gor...@gmail.com wrote:

Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to
newer versions. In particular, I tested the following with versions 1.2.4,
1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan),
it appears that the "from" parameter is being ignored.

example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied
query, so with from=1 no documents are returned.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:
c10ed0583104036a94e110f0a8b5fd7d4&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters,
but also specify a scroll parameter, we incorrectly get the single document
returned from the query.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:
c10ed0583104036a94e110f0a8b5fd7d4&from=1&scroll=2s | json

notes

Both of the above test cases work correctly in version 1.0.1

In newer versions (where example 2 fails), I noticed that the
from value is not present in the data that was returned from the
query.

I understand with with a *search_type=scan, *this would be expected
because sorting is disabled. Also, please note this is a greatly
simplified query just for illustration purposes. I've attached a sample of
an actual query at the bottom of this post, and it does contain
sorting specifications.

Is this a known issue or is this being done by design in the newer ES
versions. A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query

{
"sort": [
{
"_type": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"name_lower": {
"order": "asc",
"ignore_unmapped": true
}
},
{
"dds_key": {
"order": "asc",
"ignore_unmapped": true
}
}
],
"query": {
"filtered": {
"filter": {
"and": [
{
"or": [
{
"term": {
"parent": "
c10ed0583104012f94e11ad0ac36f2aaf"
}
}
]
},
{
"not": {
"term": {
"vcn": "DeleteMarker"
}
}
},
{
"not": {
"exists": {
"field": "notfinalized"
}
}
}
]
},
"query": {
"match_all": {}
}
}
},
"from": 1,
"size": 1000
}

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Met vriendelijke groet,

Martijn van Groningen

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c6ce78f2-cfe1-43e6-b759-2cb658881d99%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.