# Querying nested objects using Dataframes in Spark

**URL:** <https://discuss.elastic.co/t/querying-nested-objects-using-dataframes-in-spark/49958>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [May 13, 2016, 2:19am UTC](https://discuss.elastic.co/t/querying-nested-objects-using-dataframes-in-spark/49958 "2016-05-13T02:19:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cyril\_Scetbon](https://avatars.discourse-cdn.com/v4/letter/c/35a633/32.png) [@Cyril\_Scetbon](https://discuss.elastic.co/u/Cyril_Scetbon)\
**Post date:** [May 13, 2016, 2:19am UTC](https://discuss.elastic.co/t/querying-nested-objects-using-dataframes-in-spark/49958/1 "2016-05-13T02:19:41Z")

</div>

Hi,

I have the following mapping :

```
"mappings": {
    "locations": {
      "properties": {
        "addresses": {
          "type": "nested",
          "properties": {
            "id": { "type": "string" },
            "type": { "type": "string" }
          }
        }
      }
    }
  }

```

I already have a DF containing address ids, and I'd like for all of them to get the corresponding document ids by looking for the "id" in **addresses** fields.  
I've found the current solution :  
df.select(explode(df("[addresses.id](http://addresses.id)")).as("aid"), df("id"))  
.join(df\_aids, $"aid" === df\_aids("id"))  
.select(df("id"), df\_aids("id"))

I'm concerned about performance. Is it the best way to find documents in df containing in "[addresses.id](http://addresses.id)" ids from df\_aids ?

Thanks

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [May 18, 2016, 7:00am UTC](https://discuss.elastic.co/t/querying-nested-objects-using-dataframes-in-spark/49958/2 "2016-05-18T07:00:15Z")

</div>

Try to enable logging and see the resulting ES query DSL. Typically joins are handled by spark alone w/o pushdown. The 'in' filter is supported so having such a query should result in good results.

---

<div class="post-metadata">

**Author:** ![Cyril\_Scetbon](https://avatars.discourse-cdn.com/v4/letter/c/35a633/32.png) [@Cyril\_Scetbon](https://discuss.elastic.co/u/Cyril_Scetbon)\
**Post date:** [May 18, 2016, 2:47pm UTC](https://discuss.elastic.co/t/querying-nested-objects-using-dataframes-in-spark/49958/3 "2016-05-18T14:47:40Z")

</div>

but how would I do to get couples of (id, aid) using in operation. It seems isin works with a list not a DF. It seems to need more work to do it no ? it'd be great If you have a sample using my code as base  
Keep also in mind that [addresses.id](http://addresses.id) is an array

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [May 24, 2016, 10:21am UTC](https://discuss.elastic.co/t/querying-nested-objects-using-dataframes-in-spark/49958/4 "2016-05-24T10:21:31Z")

</div>

That's more of a question for Spark itself on how to translate a certain query into a range/`In` query. It's the planner in the end that decides how to translate a query (DF methods as well) into basic operations...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:24pm UTC](https://discuss.elastic.co/t/querying-nested-objects-using-dataframes-in-spark/49958/5 "2017-07-06T13:24:35Z")

</div>