# Creating Arrays of Nested Objects using the Spark Connector

**URL:** <https://discuss.elastic.co/t/creating-arrays-of-nested-objects-using-the-spark-connector/135412>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [June 11, 2018, 5:52pm UTC](https://discuss.elastic.co/t/creating-arrays-of-nested-objects-using-the-spark-connector/135412 "2018-06-11T17:52:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mike\_metistream](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@mike\_metistream](https://discuss.elastic.co/u/mike_metistream)\
**Post date:** [June 11, 2018, 5:52pm UTC](https://discuss.elastic.co/t/creating-arrays-of-nested-objects-using-the-spark-connector/135412/1 "2018-06-11T17:52:54Z")

</div>

I have a Spark DataFrame of the schema:

```
 |-- ROW_ID: string (nullable = true)
 |-- SUBJECT_ID: string (nullable = true)
 |-- HADM_ID: string (nullable = true)
 |-- CHARTDATE: string (nullable = true)
 |-- CHARTTIME: string (nullable = true)
 |-- STORETIME: string (nullable = true)
 |-- CATEGORY: string (nullable = true)
 |-- DESCRIPTION: string (nullable = true)
 |-- CGID: string (nullable = true)
 |-- ISERROR: string (nullable = true)
 |-- TEXT: string (nullable = true)
 |-- annotations: array (nullable = true)
 | |-- element: struct (containsNull = true)
 | | |-- text: string (nullable = true)
 | | |-- subject: string (nullable = true)
 | | |-- polarity: integer (nullable = false)
 | | |-- confidence: float (nullable = false)
 | | |-- historyOf: integer (nullable = false)
 | | |-- ontologyMappings: array (nullable = true)
 | | | |-- element: struct (containsNull = true)
 | | | | |-- preferredText: string (nullable = true)
 | | | | |-- codingScheme: string (nullable = true)
 | | | | |-- code: string (nullable = true)
 | | | | |-- cui: string (nullable = true)
 | | | | |-- tui: string (nullable = true)

```

I am indexing this entire structure in ElasticSearch, but neither the `annotations` field (Array of StructTypes), nor the `ontologyMappings` field are showing up as nested schemas. For example, the `ontologyMappings` mapping is shown below:

```
"ontologyMappings": {
                "properties": {
                  "code": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "codingScheme": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "cui": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "preferredText": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "code": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },

```

Is there a way to force these to be written as nested types instead of just being objects with property fields? I would like to be able to run queries that find documents that contain an instance where `code` is a particular string and associated `polarity` is 1 (under `ontologyMappings`). Without nesting, this association is impossible.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [June 21, 2018, 3:29pm UTC](https://discuss.elastic.co/t/creating-arrays-of-nested-objects-using-the-spark-connector/135412/2 "2018-06-21T15:29:50Z")

</div>

ES-Hadoop depends on the the built in Elasticsearch automatic field mapping. By default, Elasticsearch will not create nested fields when given an object. To get around this, we suggest that users set up index and mapping templates with Elasticsearch that will be picked up at index creation time.

---

<div class="post-metadata">

**Author:** ![mike\_metistream](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@mike\_metistream](https://discuss.elastic.co/u/mike_metistream)\
**Post date:** [June 21, 2018, 5:19pm UTC](https://discuss.elastic.co/t/creating-arrays-of-nested-objects-using-the-spark-connector/135412/3 "2018-06-21T17:19:33Z")

</div>

Thanks for the reply James. We ended up doing exactly that and seeding the indices we generate with the proper nested schema (eg. making the index with the specified nested fields and letting upsert handle the non-nested ones).

What the mapping seed looks like:

```
"""
{
   "mappings":{
      "data":{
         "properties":{
            "emberNLP":{
               "type":"nested",
               "properties":{
                  "ontologyMappings":{
                     "type":"nested",
                     "properties":{
                        "code":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        },
                        "codingScheme":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        },
                        "cui":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        },
                        "preferredText":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        },
                        "tui":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        }
                     }
                  }
               }
            }
         }
      }
   }
}
    """
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 19, 2018, 5:19pm UTC](https://discuss.elastic.co/t/creating-arrays-of-nested-objects-using-the-spark-connector/135412/4 "2018-07-19T17:19:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.