Pagination using search type Query_and_fetch

Elastic Stack Elasticsearch
1

According to Elastic search
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-search-type.html#query-and-fetch if
working with Query_and_fetch will return size times number of shards
hits.Following this rule I implemented below query using Nest having 5
shards

int from = 0;
int size = 1;

     while (true)
    {
        var result = client.Search<object>(x => x
       .SearchType(SearchType.QueryAndFetch)
       .Size(size)
       .From(from * size)
       .Query(qr => qr.MatchAll())
       );

        if (result.Hits.Count() == 0)
        {
            break;
        }
       from++;
    }

If I have total 17 records in index then according to the rule above query
should execute 4 times returning 5,5,5,2 = 17 total records but this is not
the case in the above query.
Above query executes 4 times returning 5,3,3,1 = 12 records missing 5
records.
Am I doing anything wrong or missing anything?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/311c56ad-c3b9-4aa7-b268-9c3c92e21e55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I think this is a bug indeed. The QUERY_AND_FETCH search mode has been
designed as an optimization of QUERY_THE_FETCH for the single shard case
(which is typically the case when you use routing) rather than a search
mode on its own.

As a side note, using QUERY_AND_FETCH when paginating on several shards
would be very bad as it would be much more I/O and memory intensive than
QUERY_THEN_FETCH. So while we could fix it to return correct results when
paginating, maybe the right fix would rather be to fix documentation to
document this search type as an internal optinmization for the single-shard
case which should not be used explicitely.

On Sat, Feb 7, 2015 at 3:58 AM, Hardik Dobariya <
hardikdobariya1132@gmail.com> wrote:

According to Elastic search
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-search-type.html#query-and-fetch if
working with Query_and_fetch will return size times number of shards
hits.Following this rule I implemented below query using Nest having 5
shards

int from = 0;
int size = 1;

     while (true)
    {
        var result = client.Search<object>(x => x
       .SearchType(SearchType.QueryAndFetch)
       .Size(size)
       .From(from * size)
       .Query(qr => qr.MatchAll())
       );

        if (result.Hits.Count() == 0)
        {
            break;
        }
       from++;
    }

If I have total 17 records in index then according to the rule above query
should execute 4 times returning 5,5,5,2 = 17 total records but this is not
the case in the above query.
Above query executes 4 times returning 5,3,3,1 = 12 records missing 5
records.
Am I doing anything wrong or missing anything?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/311c56ad-c3b9-4aa7-b268-9c3c92e21e55%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/311c56ad-c3b9-4aa7-b268-9c3c92e21e55%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j66-8r9dd6HXEPm7rcpARVutbWZOKAMUWN0ng5y6Qq3Qw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks Adrien for quick reply,

So I cannot use QUERY_AND_FETCH because this would be very expensive
regarding high I/O and memory intensive.
The following is the reason I want to use QUERY_AND_FETCH because
If I use QUERY_THEN_FETCH then according to Elastic search QUERY_AND_FETCH
is very fast because it does not sort the result on score
and QUERY_THEN_FETCH first fetch result from all shards ,perform sorting
on score then returns the result.
I do not want sorting on the result so I want to use QUERY_AND_FETCH.
Is there any other alternative for the same?
How about using scantype=scan?will it be a good practice to use it or this
will would also be expensive regarding high I/O and memory intensive

On Monday, February 9, 2015 at 12:33:56 AM UTC+5:30, Adrien Grand wrote:

I think this is a bug indeed. The QUERY_AND_FETCH search mode has been
designed as an optimization of QUERY_THE_FETCH for the single shard case
(which is typically the case when you use routing) rather than a search
mode on its own.

As a side note, using QUERY_AND_FETCH when paginating on several shards
would be very bad as it would be much more I/O and memory intensive than
QUERY_THEN_FETCH. So while we could fix it to return correct results when
paginating, maybe the right fix would rather be to fix documentation to
document this search type as an internal optinmization for the single-shard
case which should not be used explicitely.

On Sat, Feb 7, 2015 at 3:58 AM, Hardik Dobariya <hardikdob...@gmail.com
<javascript:>> wrote:

According to Elastic search
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-search-type.html#query-and-fetch if
working with Query_and_fetch will return size times number of shards
hits.Following this rule I implemented below query using Nest having 5
shards

int from = 0;
int size = 1;

     while (true)
    {
        var result = client.Search<object>(x => x
       .SearchType(SearchType.QueryAndFetch)
       .Size(size)
       .From(from * size)
       .Query(qr => qr.MatchAll())
       );

        if (result.Hits.Count() == 0)
        {
            break;
        }
       from++;
    }

If I have total 17 records in index then according to the rule above
query should execute 4 times returning 5,5,5,2 = 17 total records but this
is not the case in the above query.
Above query executes 4 times returning 5,3,3,1 = 12 records missing 5
records.
Am I doing anything wrong or missing anything?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/311c56ad-c3b9-4aa7-b268-9c3c92e21e55%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/311c56ad-c3b9-4aa7-b268-9c3c92e21e55%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b099a9c1-e870-4e50-8515-5265b035ecd2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.