# Sub queries with Spark hadoop

**URL:** <https://discuss.elastic.co/t/sub-queries-with-spark-hadoop/73189>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [January 30, 2017, 10:39am UTC](https://discuss.elastic.co/t/sub-queries-with-spark-hadoop/73189 "2017-01-30T10:39:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![shaimr](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@shaimr](https://discuss.elastic.co/u/shaimr)\
**Post date:** [January 30, 2017, 10:39am UTC](https://discuss.elastic.co/t/sub-queries-with-spark-hadoop/73189/1 "2017-01-30T10:39:06Z")

</div>

Hey, first time here, so apologies if it is a duplicate.

I use ES-Hadoop to run over all index data, but I need to perform sub-queries inside my mapPartition.  
I query for entire index, then run over it with mapPartition, but I need to perform another query for each item from other index.

For example, I have an index of dogs and another index of dogs-relations (their band). The first index of dogs is the index that I read with ES-Hadoop as RDD and I need to query the second query for each dog separately.

How can I solve this without perform a query per dog?  
I want to narrow queries number as possible, but I don't know how to merge those RDDs in a smart way.

BTW - I opened a [StackOverflow question](http://stackoverflow.com/questions/41552208/how-to-join-rdds-based-on-elastic-hadoop)

Any Idea will help,  
Thanks.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [February 2, 2017, 7:46pm UTC](https://discuss.elastic.co/t/sub-queries-with-spark-hadoop/73189/2 "2017-02-02T19:46:22Z")

</div>

@shaimr you could always try a join operation in Spark using RDD's sourced from both indices.

---

<div class="post-metadata">

**Author:** ![shaimr](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@shaimr](https://discuss.elastic.co/u/shaimr)\
**Post date:** [February 5, 2017, 8:00am UTC](https://discuss.elastic.co/t/sub-queries-with-spark-hadoop/73189/3 "2017-02-05T08:00:38Z")

</div>

The first index contains 50M documents and the relation index contains 300M documents,  
Isn't it too much to perform memory join?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 5, 2017, 8:01am UTC](https://discuss.elastic.co/t/sub-queries-with-spark-hadoop/73189/4 "2017-03-05T08:01:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.