Max size for getting data

Elastic Stack Elasticsearch
1

Hi

What is the max size for getting data?

get /pupil/_search
{
      "from": 0,
      "size": 10000,
       "query":{
         "match_all": {}
        }                                           
}

I have more than 70000 data. But is showing

{
  "error": {
    "root_cause": [
      {
        "type": "query_phase_execution_exception",
        "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "pupil",
        "node": "EjNcgN2HQWW6P42Wur-dxw",
        "reason": {
          "type": "query_phase_execution_exception",
          "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
        }
      }
    ]
  },
  "status": 500
}

How can I solve this issue?

Thanks
Aneesh

2

Use scroll API if you need to export data.

3

Hi David Pilato

Unsuccessful low level call on POST: /pupil/pupil/_search
# Audit trail of this API call:
 - [1] BadResponse: Node: http://localhost:9200/ Took: 00:00:00.0521511
# ServerError: ServerError: 500Type: search_phase_execution_exception Reason: "all shards failed"
# Request:

                                {
                                    "from": 0,
                                    "size": 100000,
                                    "query" : {
                                    "bool": {
                                                "must_not": [
                                                    {
                                                        "match": {
                                                            "_id": "null"
                                                                    }
                                                    },
                                                    {
                                                        "match": {
                                                            "_id": "-1"
                                                                    }
                                                    }
                                                    ,
                                                    {
                                                        "match": {
                                                            "_id": "lastmodified"
                                                                    }
                                                    }
                                                                ]
                                                }
                                            }
                                                                    
                                }
# Response:
{"error":{"root_cause":[{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"pupil","node":"EjNcgN2HQWW6P42Wur-dxw","reason":{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}]},"status":500}

Nest Query is giving error. Scroll API will help in this?

Thanks
Aneesh L

4

It's all written in the error

See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

5

Hi David Pilato

I changed settings, It worked

Thanks
Aneesh

6

I hope you understand the consequences of changing settings.
Specifically in term of memory pressure.

7

Hi David Pilato

I need to get 100000 records. Is there any other solution for getting big data, without changing index settings.

Thanks
Aneesh L

8

Yes. Scroll API.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.