Fetch command in elasticsearch queries, limit query results

[liron_gofberg]

Cristian, please

I don’t care how much time it will takes
I just want to do it the right way
The optimal way

I’m using only filters term queries
All my queries are similar
And all of them return millions of records
I just want to let the user fetch millions of records and to let him decide how much results he wants to get out of the full result

Each document has 40 fields , I want to getch them all

1.what is the right solution to fetch millions of records?
2.how can I fetch only specific number of results out of the full result ?

If user wants more than 10000 results, you don’t have a feature to get just part of the results? So user will get accurate specific number of records out of the whole search result?(when results number is bigger than 10000)

I would like to get a low latency
But if it’s not possible to do it fast it’s ok
I just want to know how to do it

Terminate_after doesn’t help to get accurate number of results , its operation is on the shard level

You tell me that scroll don’t need to fetch such amount of data do
So what can do that?

Concurrency : max 30
And most of the time much less than 30
No updates
Deletion data will be rare
Indexing rate - few millions per a day

I saw multiple posts talking about fetching millions of records but no one tells how
And no one tells how to get accurate results number out of the full result

Is search after is the right solution for fetching millions of records?
Scroll ?

[dadoonet (David Pilato)]

You can use:

• the size and from parameters to display by default up to 10000 records to your users. If you want to change this limit, you can change index.max_result_window setting but be aware of the consequences (ie memory).
• the search after feature to do deep pagination.
• the Scroll API if you want to extract a resultset to be consumed by another tool later.

[liron_gofberg]

Thanks,
so search after is the right solution to get millions of results
Then I can insert the results into data frame

It will be optimal if I will configure Elasticsearch to sort the documents?
I saw example of reading 1 page only
Do you have example of reading 2 pages or more?
Please🙏🏻

Thanks