Greetings,
We have tons of data coming into our ES from devices across the whole US. We need to do a lot of analytical research on this data. We do not want to do this on the live data but would like to do it on copies of it. What would be the best approach of getting this data into another node (research node) separate from the production node on a daily basis. I have already tried a search option as follows
result = elastic_client.search(index="slushies_*", body=search_param)
response_hits = result["hits"]["hits"]
and then looping through the result set. This is not scalable and it takes a long time to go through a days worth of data.
Is there other recommended options, such as restoring backups, or some other sort of snapshots?
Thanks!