Searches using Search_after timeout after three pages

Hello,

I have about 117 million records in Elasticsearch and I am trying to fetch 2.5 million records using pagination. I tried to use the Scroll API and the Search_After. Since I want to fetch such huge number of records frequently, I ruled out Scroll API as my best option. I am using elastic search v6.4

Using search_after, I have been seeing that requests time out consistently after three pages and the time taken for ES search queries is much higher too. What is the ideal way to use search_after to fetch millions of records? And can these search_after queries be issued in parallel?
Here is what my sample query looks like-

POST /indexname/_search
{
  "size": 10000,
  "sort": [
    {
      "docid": {
        "order": "asc"
      },
      "date": {
        "order": "asc"
      }
    }
  ],
  "search_after": [
          "02073930-F6E8-47D8-8E72-72F335B79F09",
          1539820800000
        ],
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [
              {
                "term": {
                  "testId": {
                    "value": 1540401
                  }
                }
              },
              {
                "term": {
                  "testCode": {
                    "value": "0bfa81a6-9e1e-437b-9600-bcf22d50781a"
                  }
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must_not": [
              {
                "terms": {
                  "testStatus": [
                    "CANCELLED",
                    "REJECTED"
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.