# Error load as a DataFrame **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [February 3, 2016, 8:58am UTC](https://discuss.elastic.co/t/error-load-as-a-dataframe/40842 "2016-02-03T08:58:33Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![rss87](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rss87](https://discuss.elastic.co/u/rss87)\ **Post date:** [February 3, 2016, 8:58am UTC](https://discuss.elastic.co/t/error-load-as-a-dataframe/40842/1 "2016-02-03T08:58:33Z")
I try to load a index as a DF with this command : val sqlc = new SQLContext(sc) val usersDF = sqlc.esDF("users-2016/person") after that, I register this as a TempTable --\> usersDF.registerTempTable("users") and I try to execute a simple query: val result =sqlc.sql("SELECT count(\*) FROM users") result.show() but.... I always get the same error: java.util.NoSuchElementException : key not found : name with usersDF.printSchema() return this: root |-- @timestamp: timestamp (nullable = true) |-- @version: string (nullable = true) |-- dst\_ip: string (nullable = true) |-- name: string (nullable = true) |-- request: string (nullable = true) |-- last\_name: string (nullable = true) The index has 8 millions of documents How I can solve this problems? ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [February 21, 2016, 5:58pm UTC](https://discuss.elastic.co/t/error-load-as-a-dataframe/40842/2 "2016-02-21T17:58:55Z")
What version of ES-Hadoop are you using and what version of Spark? Also what does your mapping in ES looks like? Also what's full strack trace that you get? ---
**Author:** ![rss87](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rss87](https://discuss.elastic.co/u/rss87)\ **Post date:** [February 21, 2016, 9:00pm UTC](https://discuss.elastic.co/t/error-load-as-a-dataframe/40842/3 "2016-02-21T21:00:11Z")
first, thanks for answering, the Spark version is 1.4.2 and ES-Hadoop is 2.2.0-m1. I think that the problem is due to not all fields are present in the elasticsearch Index, for example the field "name" is not present in 500k documents ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [February 21, 2016, 10:05pm UTC](https://discuss.elastic.co/t/error-load-as-a-dataframe/40842/4 "2016-02-21T22:05:38Z")
Please use es-hadoop version 2.2 GA. A lot of things have been fixed vs m1 (which was a milestone). ---
**Author:** ![rss87](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rss87](https://discuss.elastic.co/u/rss87)\ **Post date:** [February 22, 2016, 9:48am UTC](https://discuss.elastic.co/t/error-load-as-a-dataframe/40842/5 "2016-02-22T09:48:52Z")
HI, I've used 2.2.0 GA and now I get this error: 16/02/22 10:44:12 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) org.elasticsearch.hadoop.EsHadoopIllegalStateException: Field 'positions.start\_date' not found; typically this occurs with arrays which are not mapped as single value at org.elasticsearch.spark.sql.RowValueReader$class.rowColumns(RowValueReader.scala:34) at org.elasticsearch.spark.sql.ScalaRowValueReader.rowColumns(ScalaEsRowValueReader.scala:14) at org.elasticsearch.spark.sql.ScalaRowValueReader.createMap(ScalaEsRowValueReader.scala:42) at org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:766) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:692) at org.elasticsearch.hadoop.serialization.ScrollReader.list(ScrollReader.java:734) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:687) at org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:794) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:692) at org.elasticsearch.hadoop.serialization.ScrollReader.readHitAsMap(ScrollReader.java:457) at org.elasticsearch.hadoop.serialization.ScrollReader.readHit(ScrollReader.java:382) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:277) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:250) at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:456) at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:86) at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:43) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) at [scala.collection.TraversableOnce$class.to](http://scala.collection.TraversableOnce%24class.to)(TraversableOnce.scala:273) at [scala.collection.AbstractIterator.to](http://scala.collection.AbstractIterator.to)(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) at org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215) at org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [February 22, 2016, 6:43pm UTC](https://discuss.elastic.co/t/error-load-as-a-dataframe/40842/6 "2016-02-22T18:43:25Z")
Repeating myself, what is your mapping in ES? What does the docs look like? Do they contain arrays (as the exception message indicates)? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:26pm UTC](https://discuss.elastic.co/t/error-load-as-a-dataframe/40842/7 "2017-07-06T13:26:05Z")