# ES-Spark : issue with 'read\_field\_include' in case of nested objects

**URL:** <https://discuss.elastic.co/t/es-spark-issue-with-read-field-include-in-case-of-nested-objects/47779>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [April 19, 2016, 11:18am UTC](https://discuss.elastic.co/t/es-spark-issue-with-read-field-include-in-case-of-nested-objects/47779 "2016-04-19T11:18:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Preeti\_Raj\_Buchhada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/preeti_raj_buchhada/32/8971_2.png) [@Preeti\_Raj\_Buchhada](https://discuss.elastic.co/u/Preeti_Raj_Buchhada)\
**Post date:** [April 19, 2016, 11:18am UTC](https://discuss.elastic.co/t/es-spark-issue-with-read-field-include-in-case-of-nested-objects/47779/1 "2016-04-19T11:18:03Z")

</div>

A typical record in my ES index looks like:

```
"_source": {
      "app": "panoply",
      "response": {
         "category": "uncategorized",
         "subcategory": "uncategorized",
         "activity_common_name": "name123",
         "score": 0,
         "duration_secs": 2,
         "sub_activity": null,
         "activity": "name123"
      },
      "member_id": 2357919,
      "device_user_identity": 1688734,
      "activity_type": "type123",
      "response_timestamp": "2016-01-10T23:05:18.000Z"
   }

```

When I created a TABLE using Spark Shell as follows:

```
sql("""
      CREATE TEMPORARY TABLE jan10
      USING org.elasticsearch.spark.sql
      OPTIONS (
        resource 'cortez/data',
        nodes 'localhost',
        port '9201',
        scroll_size '500',
        query '?response_timestamp:[2016-01-01 TO 2016-01-10]',
        read_field_include 'member_id,response.category,response.subcategory,response.activity,response.activity_common_name,response.duration_secs,response.sub_activity,response_timestamp'
      ) """)

**Note:** response.score is not included in 'read_field_include'

```

and executed

`sql("""SELECT * from jan10""").show()`

I observed that **all field values after response.score (namely duration\_secs, sub\_activity and activity) are showing up as null.**  
If I add response.score to 'read\_field\_include', all filed values are fetched correctly.

Seems like a bug.  
Can you please check.  
Thanks.

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [April 21, 2016, 6:06am UTC](https://discuss.elastic.co/t/es-spark-issue-with-read-field-include-in-case-of-nested-objects/47779/2 "2016-04-21T06:06:54Z")

</div>

It looks like a bug that probably triggers skipping of nested fields.  
What version of ES-Hadoop are you using?

Cheers,

---

<div class="post-metadata">

**Author:** ![Preeti\_Raj\_Buchhada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/preeti_raj_buchhada/32/8971_2.png) [@Preeti\_Raj\_Buchhada](https://discuss.elastic.co/u/Preeti_Raj_Buchhada)\
**Post date:** [April 21, 2016, 6:40am UTC](https://discuss.elastic.co/t/es-spark-issue-with-read-field-include-in-case-of-nested-objects/47779/3 "2016-04-21T06:40:47Z")

</div>

Environment:  
**ES** : 1.3.2  
**es-hadoo** p: elasticsearch-hadoop-2.3.0  
**Spark:** spark-1.6.1-bin-hadoop2.6

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:24pm UTC](https://discuss.elastic.co/t/es-spark-issue-with-read-field-include-in-case-of-nested-objects/47779/4 "2017-07-06T13:24:58Z")

</div>