Restriction on amount of records returned by elasticsearch

[bilpor (william crawley)]

Hi All,

It doesn't matter what type of search I perform. (I know for example one particular search I do should return over 1000 records/entries on the index). In Kibana using the Dev Tools window, the output window may show hit's with a total 1033 but actual records returned are never any greater than 10. I'm trying to perform the exact same query via classic asp and a call using the MSXML2 library and this has the same issue. It returns me the same records that I see via Kibana,

So I'm thinking that there is a restriction somewhere in the ElasticSearch config, but I dont know what to look for to increase this size.

[dadoonet (David Pilato)]

"size": 1000 ?

[bilpor (william crawley)]

Basically if I do my select in SQL Server. i have returned 1033 records. The same index that indexes that table will show the hits total 1033, but will only return me 10 records. i.e. Im expecting elasticsearch to return me 1033 documents, but it never returns more than 10. Even when I just do a simple match_all I still get only 10

in SQL if I do

select * from News

it will return 1737 rows

in Kibana against the index that maps to this table if I do:

GET newsindex/_search
{
     "query": { "match_all" : {}}
}

then in the kibana output window I see

.... "hits": { "total": 1737, .....

but paging down the window I dont see anymore than 10.

For further clarity:

when I created the index, I created a property called "author". I only see 10 "author" values

[dadoonet (David Pilato)]

So I said, run:

GET newsindex/_search
{
  "size": 1737
}

And you'll get your 1737 hits.

[bilpor (william crawley)]

that's ok in my test scenario, but for real I may not know the amount of records I'm expecting back, I need the query to send me back ALL available records based on my Query not just the first 10 or a subset.

I also just ran your code, and it still only returned me 10 entries in the kibana output window

[dadoonet (David Pilato)]

I also just ran your code, and it still only returned me 10 entries in the kibana output window

I don't know. But yeah may be.

I need the query to send me back ALL available records based on my Query not just the first 10 or a subset.

Then you need to use scroll API which is built for that: Scroll | Elasticsearch Reference [5.6] | Elastic

[bilpor (william crawley)]

HI dadoonet,

I tried an example using this link. On my initial search it did return more than 10 records, and it returned me a scroll Id. I then did a get using the scroll id based on the documentation i.e.:

GET _search/scroll
{
  "scroll": "1m",
  "scroll_id": "DnF1ZXJ5VGhlbkZldGNoAwAAAAAACmZTFlcwMm9YN3NlUjZXMnBxTDZnX2E4aEEAAAAAAApmURZXMDJvWDdzZVI2VzJwcUw2Z19hOGhBAAAAAAAKZlIWVzAyb1g3c2VSNlcycHFMNmdfYThoQQ=="
}

and it always errors of the type: "search_context_missing_exception"

I first thought perhaps the first get returned me all my records hence, nothing left to scroll, so I reduced the size parameter on my inital query, and tried again after deleting the scroll. This still made no difference.

[dadoonet (David Pilato)]

Can you reproduce with a full example, including the top of the responses?