# Effiecient way to store the result of a large slow query

**URL:** <https://discuss.elastic.co/t/effiecient-way-to-store-the-result-of-a-large-slow-query/18227>\
**Category:** Elasticsearch\
**Created:** [June 20, 2014, 7:58am UTC](https://discuss.elastic.co/t/effiecient-way-to-store-the-result-of-a-large-slow-query/18227 "2014-06-20T07:58:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chen\_Wang](https://avatars.discourse-cdn.com/v4/letter/c/b5a626/32.png) [@Chen\_Wang](https://discuss.elastic.co/u/Chen_Wang)\
**Post date:** [June 20, 2014, 7:58am UTC](https://discuss.elastic.co/t/effiecient-way-to-store-the-result-of-a-large-slow-query/18227/1 "2014-06-20T07:58:56Z")

</div>

Hi guys,  
Just wondering what is the most efficient way of executing a query that  
takes time(parent/child documents) and returns large amount of entries, and  
store the result in randomly evenly divided block to HDFS? e.g, the query  
will return 100million records and I want every random 1million stored in a  
different location(file/folder) on HDFS.

I assume I could execute the query with scroll, and then whenever I  
received the 1 million records back, I then spawn anther thread to commit  
it to HDFS? Is there a way to run the query distributed way and have 100  
threads query ES at the same time and each getting a random 1million  
back(without duplicate)? will ES hadoop help in this case?

Appreciate your input!  
Chen

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACim9Rm64uHE9EQ35r\_mJr9VhiEbDfD-70vS1uQHSG6UXM7ZDQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACim9Rm64uHE9EQ35r_mJr9VhiEbDfD-70vS1uQHSG6UXM7ZDQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:20am UTC](https://discuss.elastic.co/t/effiecient-way-to-store-the-result-of-a-large-slow-query/18227/2 "2017-07-06T01:20:59Z")

</div>

Closed on July 6, 2017, 1:20am UTC