# Reading Nested data from ElasticSearch via Spark Scala

**URL:** <https://discuss.elastic.co/t/reading-nested-data-from-elasticsearch-via-spark-scala/221403>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [February 28, 2020, 10:36am UTC](https://discuss.elastic.co/t/reading-nested-data-from-elasticsearch-via-spark-scala/221403 "2020-02-28T10:36:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Apurw](https://avatars.discourse-cdn.com/v4/letter/a/90db22/32.png) [@Apurw](https://discuss.elastic.co/u/Apurw)\
**Post date:** [February 28, 2020, 10:36am UTC](https://discuss.elastic.co/t/reading-nested-data-from-elasticsearch-via-spark-scala/221403/1 "2020-02-28T10:36:06Z")

</div>

I am trying to read data from Elasticsearch via Spark Scala:

Scala 2.11.8, Spark 2.3.0, Elasticsearch 5.6.8

To Connect -- `spark2-shell --jars elasticsearch-spark-20_2.11-5.6.8.jar`

```auto
val df = spark.read.format("org.elasticsearch.spark.sql").option("es.nodes", "xxxxxxx").option("es.port", "xxxx").option("es.net.http.auth.user","xxxxx").option("spark.serializer", "org.apache.spark.serializer.KryoSerializer").option("es.net.http.auth.pass", "xxxxxx").option("es.net.ssl", "true").option("es.nodes.wan.only", "true").option("es.net.ssl.cert.allow.self.signed", "true").option("es.net.ssl.truststore.location", "xxxxx").option("es.net.ssl.truststore.pass", "xxxxx").option("es.read.field.as.array.include","true").option("pushdown", "true").option("es.read.field.as.array.include","a4,a4.a41,a4.a42,a4.a43,a4.a43.a431,a4.a43.a432,a4.a44,a4.a45").load("<index_name>") 

```

Schema as below

```auto
 |-- a1: string (nullable = true)
 |-- a2: string (nullable = true)
 |-- a3: struct (nullable = true)
 | |-- a31: integer (nullable = true)
 | |-- a32: struct (nullable = true)
 |-- a4: array (nullable = true)
 | |-- element: struct (containsNull = true)
 | | |-- a41: string (nullable = true)
 | | |-- a42: string (nullable = true)
 | | |-- a43: struct (nullable = true)
 | | | |-- a431: string (nullable = true)
 | | | |-- a432: string (nullable = true)
 | | |-- a44: string (nullable = true)
 | | |-- a45: string (nullable = true)
 |-- a8: string (nullable = true)
 |-- a9: array (nullable = true)
 | |-- element: struct (containsNull = true)
 | | |-- a91: string (nullable = true)
 | | |-- a92: string (nullable = true)
 |-- a10: string (nullable = true)
 |-- a11: timestamp (nullable = true)

```

Though I am able to read data from direct columns and nested schema level 1 (i.e a9 or a3 columns) via command:

```auto
df.select(explode($"a9").as("exploded")).select("exploded.*").show

```

Problem is occuring when I am trying to read a4 elements as its throwing me below error:

```auto
: scala.MatchError: Buffer() (of class scala.collection.convert.Wrappers$JListWrapper)
        at org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:276)
        at org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:275)
        at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:103)
        at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:241)
        at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:231)
        at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:103)
        at org.apache.spark.sql.catalyst.CatalystTypeConverters$ArrayConverter$anonfun$toCatalystImpl$2.apply(CatalystTypeConverters.scala:164)
        at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234)............

```

Anything I am doing wrong or any steps I am missing? Please Help

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 27, 2020, 10:36am UTC](https://discuss.elastic.co/t/reading-nested-data-from-elasticsearch-via-spark-scala/221403/2 "2020-03-27T10:36:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.