Query and return all of the results

Elastic Stack Elasticsearch
1

I'm trying to query elasticsearch to return all the index data between say july 3 and july 4.

I issued the query and it returned a total hits of 14003. I know the default value it returns is 10, and i can set the default settings up to 10000 being returned.

But If the total hits is 14003 and the max return value is only 10000, how could i get the remaining 4003 values.

I tried using the from/size: so i tried "from" : 10001 , "size" : 14003

but got the error stating that the size was too big.

Any ideas?

2

You can use the scroll API.

3

Thanks, i tried the scroll and i can't get it to return all of it as well. i can't get it to return a large number of results or even the all the results as mentioned in the document. I get pages of it. I want everything in one shot. Maybe i don't have the correct syntax

curl -XGET '10.40.163.67:9200/sn_index_data/sn-node2/_search?scroll=10m&pretty' -d '{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"range": {
"audit_info.last_accessed": {
"gte": "2016-06-29T15:00:00",
"lte": "2016-06-29T16:00:00"
}
}
}
}
}
}
'

After running the initial query, i see (below), so i'm expecting 905 indices

"hits" : {
"total" : 905,

Then after i get the scroll id i call

curl -XGET '10.40.163.67:9200/_search/scroll' -d'
{
"scroll" : "10m",
"scroll_id" : "cXVlcnlUaGVuRmV0Y2g7NTs2NzkxOjBCYURadjJVVDZhTG5heVJndzEyV1E7Njc5MDowQmFEWnYyVVQ2YUxuYXlSZ3cxMldROzI0OTgxNDpTVTR1RXgwUVNXcVJXTV9HOVpCdlZnOzI0OTgxNTpTVTR1RXgwUVNXcVJXTV9HOVpCdlZnOzI0OTgxNjpTVTR1RXgwUVNXcVJXTV9HOVpCdlZnOzA7"
}
'

4

Figured it out.

curl -XGET '10.40.163.67:9200/sn_index_data/sn-node2/_search?scroll=10m&size=905' -d '{

does the trick.

5

It is inefficient to just set the size to something huge. 905 isn't huge.
For that many docs you don't need to scroll. If you had tens of thousands
you'd need to scroll. You still set size to 1000 or 5000 or something, but
you have to deal with the pages of results.

Elasticsearch doesn't support streaming the results back which is the only
way you'd be able to request and get them all back in one response.