Download 200k records from Elasticsearch

Elastic Stack Elasticsearch
1

Hi Everyone,

We have built custom analytics UI (React) and using Elasticsearch as backend for SQL queries/analysis.

We have Export results button that exports the queried dataset. The requirement is to download upto 200k records. ES has a limitation to download only 10k out of the box and is not recommended to increase the limits more than that due to memory issues.

We are using ES _sql api for queries and results. Need your help in implementing a robust solution.

I have checked scroll api and search api options and looks like its complicated and I am unable to visualize on how to concat and download the final file.

Appreciate your help.

Thanks,
Sri

2

Hi Everyone,

Please provide your suggestions on this topic.

Thanks,
Sri

3

What have you tried and how are you making queries?

You will need to paginate your requests according to the documentation , read the part about search_after.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.