Best way to export elastic index to json / local file format

[tom_es (Tom Vin)]

Hi,
we would like to export elastic index into local file (or by rest), to some kind of format (for example, json).

our index contains million of documents (~1 GB size).
is there a recommended way for doing this using some kind of API?

fetching the data by scrolling have it's own limitations (the number of docs to fetch is limited by 10K by default) - so it's less of an option, there is also the search After API - but we have some limitations that are currently prevent using it.

our final goal is to create a script that will fetch the data into a local file.

any help will be appreciated,
Thanks,
Tom

[spinscale (Alexander Reelsen)]

Can you explain why scroll searching is a problem? Your use-case is exactly what scroll has been made for, you go step by step through all of your data without suffering the usual problems of deep pagination. You just fetch a few hundred/thousand entries per scroll and then move on page-by-page.

See https://www.elastic.co/guide/en/elasticsearch/reference/7.4/search-request-body.html#request-body-search-scroll