# Exporting sub set of data from main index to a new index **URL:** \ **Category:** Elasticsearch\ **Created:** [October 15, 2014, 5:14pm UTC](https://discuss.elastic.co/t/exporting-sub-set-of-data-from-main-index-to-a-new-index/20269 "2014-10-15T17:14:54Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Vijay\_Tiwary](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@Vijay\_Tiwary](https://discuss.elastic.co/u/Vijay_Tiwary)\ **Post date:** [October 15, 2014, 5:14pm UTC](https://discuss.elastic.co/t/exporting-sub-set-of-data-from-main-index-to-a-new-index/20269/1 "2014-10-15T17:14:54Z")
Is there a easy way to export a some part of data(based on some filters) from a index say for e.g from a master index to a new index. Apparently it looks like I will have to use bulk API to query the data from the master index (using some filters) and then I will have insert those documents into the new index. Is there any better and easier way. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f3778121-8019-44ba-a30c-3194ae72f72f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f3778121-8019-44ba-a30c-3194ae72f72f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Alexandre\_Rafalovitc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexandre_rafalovitc/32/1124_2.png) [@Alexandre\_Rafalovitc](https://discuss.elastic.co/u/Alexandre_Rafalovitc)\ **Post date:** [October 15, 2014, 6:22pm UTC](https://discuss.elastic.co/t/exporting-sub-set-of-data-from-main-index-to-a-new-index/20269/2 "2014-10-15T18:22:50Z")
Have you looked at Scroll and Scan? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. This assumes your \_source field has not been disabled. Regards, Alex. Personal: [http://www.outerthoughts.com/](http://www.outerthoughts.com/) and @arafalov Solr resources and newsletter: [http://www.solr-start.com/](http://www.solr-start.com/) and @solrstart Solr popularizers community: [Sign Up | LinkedIn](https://www.linkedin.com/groups?gid=6713853) On 15 October 2014 13:14, Vijay Tiwary [vijaykr.tiwary@gmail.com](mailto:vijaykr.tiwary@gmail.com) wrote: > Is there a easy way to export a some part of data(based on some filters) > from a index say for e.g from a master index to a new index. Apparently it > looks like I will have to use bulk API to query the data from the master > index (using some filters) and then I will have insert those documents into > the new index. Is there any better and easier way. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/f3778121-8019-44ba-a30c-3194ae72f72f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f3778121-8019-44ba-a30c-3194ae72f72f%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEFAe-HhSy0eoUg6gYdNYir%2BaBCy1L7QPqOcqdmOQ9JU2VJVkQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEFAe-HhSy0eoUg6gYdNYir%2BaBCy1L7QPqOcqdmOQ9JU2VJVkQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Vijay\_Tiwary](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@Vijay\_Tiwary](https://discuss.elastic.co/u/Vijay_Tiwary)\ **Post date:** [October 16, 2014, 12:36pm UTC](https://discuss.elastic.co/t/exporting-sub-set-of-data-from-main-index-to-a-new-index/20269/3 "2014-10-16T12:36:59Z")
Thanks Alex. I have done it like this: while (true) { scrollResp = _client.prepareSearchScroll(scrollResp.getScrollId()).setScroll(new TimeValue(600000)).execute().actionGet();_ SearchHits sh= scrollResp.getHits(); SearchHit searchHit = sh.getHits(); LOG.info("Hits :"+sh.getTotalHits()+", Docs fetched :"+searchHit.length); if (scrollResp.getHits().getHits().length == 0) { break; } } So assuming I have only on shard and i am fetching total of 100,000 documents in steps of 10,000 documents then there will be 10 get calls and this call will happen serially one after the other. Is there a mechanism by any chance to execute this get calls in parallel. On Wednesday, October 15, 2014 11:53:38 PM UTC+5:30, Alexandre Rafalovitch wrote: > Have you looked at Scroll and Scan? > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/master/scan-scroll.html) > > This assumes your \_source field has not been disabled. > > Regards, > Alex. > Personal: [http://www.outerthoughts.com/](http://www.outerthoughts.com/) and @arafalov > Solr resources and newsletter: [http://www.solr-start.com/](http://www.solr-start.com/) and @solrstart > Solr popularizers community: [Sign Up | LinkedIn](https://www.linkedin.com/groups?gid=6713853) > > On 15 October 2014 13:14, Vijay Tiwary \<[vijaykr...@gmail.com](mailto:vijaykr...@gmail.com) \\> > wrote: > > > Is there a easy way to export a some part of data(based on some filters) > > from a index say for e.g from a master index to a new index. Apparently > > it > > looks like I will have to use bulk API to query the data from the master > > index (using some filters) and then I will have insert those documents > > into > > the new index. Is there any better and easier way. > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/f3778121-8019-44ba-a30c-3194ae72f72f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f3778121-8019-44ba-a30c-3194ae72f72f%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5bce72bf-3400-4963-9fa7-adf48d12383d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5bce72bf-3400-4963-9fa7-adf48d12383d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 16, 2014, 11:10pm UTC](https://discuss.elastic.co/t/exporting-sub-set-of-data-from-main-index-to-a-new-index/20269/4 "2014-10-16T23:10:59Z")
Scan/scroll, when performed over multiple nodes/shards, is inherently executed in parallel. Scan/scroll depends on the scroll ID chain, so you have to execute a scan/scroll sequence serially. For a single shard you can add filters to the query in order to partition the search hits, these queries can be executed with several scan/scroll requests in parallel from your client. Jörg On Thu, Oct 16, 2014 at 2:36 PM, Vijay Tiwary [vijaykr.tiwary@gmail.com](mailto:vijaykr.tiwary@gmail.com) wrote: > Thanks Alex. I have done it like this: > > while (true) { > scrollResp = _client.prepareSearchScroll(scrollResp.getScrollId()).setScroll(new > TimeValue(600000)).execute().actionGet();_ > SearchHits sh= scrollResp.getHits(); > SearchHit searchHit = sh.getHits(); > > LOG.info("Hits :"+sh.getTotalHits()+", Docs fetched :"+searchHit.length); > if (scrollResp.getHits().getHits().length == 0) { > break; > } > } > > So assuming I have only on shard and i am fetching total of 100,000 > documents in steps of 10,000 documents then there will be 10 get calls and > this call will happen serially one after the other. Is there a mechanism by > any chance to execute this get calls in parallel. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE\_xwffAzsiACRB374rn6GR3hN44rnJvHfa4GZiWMxryg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_xwffAzsiACRB374rn6GR3hN44rnJvHfa4GZiWMxryg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:55am UTC](https://discuss.elastic.co/t/exporting-sub-set-of-data-from-main-index-to-a-new-index/20269/5 "2017-07-06T00:55:39Z")