We have built custom analytics UI (React) and using Elasticsearch as backend for SQL queries/analysis.
We have Export results button that exports the queried dataset. The requirement is to download upto 200k records. ES has a limitation to download only 10k out of the box and is not recommended to increase the limits more than that due to memory issues.
We are using ES _sql api for queries and results. Need your help in implementing a robust solution.
I have checked scroll api and search api options and looks like its complicated and I am unable to visualize on how to concat and download the final file.
Appreciate your help.