# Spark SQL and complex types

**URL:** https://discuss.elastic.co/t/spark-sql-and-complex-types/24873
**Category:** Elasticsearch
**Created:** [July 3, 2015, 2:54pm UTC](https://discuss.elastic.co/t/spark-sql-and-complex-types/24873 "2015-07-03T14:54:57Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![arthur1](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@arthur1](https://discuss.elastic.co/u/arthur1)
#### Post date: [July 3, 2015, 2:54pm UTC](https://discuss.elastic.co/t/spark-sql-and-complex-types/24873/1 "2015-07-03T14:54:58Z")

</div>

I am using Spark 1.3.1 to access some twitter data stored in ElasticSearch. I am using Spark SQL. The problem I am having has to do with Complex Types. For example, the following SQL works but I can't seem to retrieve the content, "SELECT twitterstatus.user.name FROM twitterstatus".

Both approaches below work if complex types are NOT referenced.  
DataFrame dataFrame = sqlContext.sql("SELECT twitterstatus.user.name FROM twitterstatus");  
List aIP = dataFrame.collectAsList();

///////////////////////////  
DataFrame dataFrame = sqlContext.sql("SELECT twitterstatus.user.name FROM twitterstatus");  
List aIP = dataFrame.toJavaRDD().map(new Function\<Row, Object\>() {  
@Override  
public Object call(Row row) {  
return row;  
}  
}).collect();

But when complex types are referenced, I see the following exception "java.lang.ClassCastException: scala.collection.mutable.LinkedHashMap cannot be cast to org.apache.spark.sql.Row".

Do you have a suggestion for retrieving these complex types from ElasticSearch?

Arthur1

---

<div class="post-metadata">

### Author: ![nolangrace](https://avatars.discourse-cdn.com/v4/letter/n/a5b964/32.png) [@nolangrace](https://discuss.elastic.co/u/nolangrace)
#### Post date: [July 9, 2015, 1:20pm UTC](https://discuss.elastic.co/t/spark-sql-and-complex-types/24873/2 "2015-07-09T13:20:05Z")

</div>

I have noticed this same thing. Is this the recommended way to access nested Elastisearch data via spark or am I on the wrong track. Would someone mind giving me some assistance?

---

<div class="post-metadata">

### Author: ![softnado](https://avatars.discourse-cdn.com/v4/letter/s/e47774/32.png) [@softnado](https://discuss.elastic.co/u/softnado)
#### Post date: [July 15, 2016, 4:30am UTC](https://discuss.elastic.co/t/spark-sql-and-complex-types/24873/3 "2016-07-15T04:30:08Z")

</div>

Maybe little bit late. But try use esJsonRDD with more code way can handle this problem. I guess somehow the Map[String,AnyRef] cause it use LinkedHashMap.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:35pm UTC](https://discuss.elastic.co/t/spark-sql-and-complex-types/24873/4 "2017-07-05T22:35:21Z")

</div>