Getting documents from an index with more than 10,000 records

Elastic Stack Elasticsearch
1

I am trying to get a list of the ids of all document in my index. The index has about 50k documents in it, however I cannot use the from parameter to see any ids once "from" + "size" > 10,000. I know there's the scroll api, but when I tried to use it, I get the error message: from is not allowed in scroll context. How can I modify my query to get the id of records past record 10,000 in my index? Thanks!

    GET vs-imr-cases/_search?scroll=1m
    {
      "query": {
        "match_all": {}
      },
      "sort": [
        {
          "case_number.keyword": {
            "order": "desc"
          }
        }
      ], 
      "_source": "_id",
      "size": 10,
      "from": 10000
    }
2

You can use:

  • the size and from parameters to display by default up to 10000 records to your users. If you want to change this limit, you can change index.max_result_window setting but be aware of the consequences (ie memory).
  • the search after feature to do deep pagination.
  • the Scroll API if you want to extract a resultset to be consumed by another tool later.
3

Indeed : remove the from/size parameters when you call the scroll API.

4

Thanks for the follow up. How do you change index.max_result_window setting? Is that something you can do after an index is created and populated?

5

I think you can. But please understand that it's a good default safeguard.

May be you could explain what you are exactly trying to achieve.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.