# Inner\_hits with elasticsearch-hadoop

**URL:** <https://discuss.elastic.co/t/inner-hits-with-elasticsearch-hadoop/245580>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [August 19, 2020, 10:09am UTC](https://discuss.elastic.co/t/inner-hits-with-elasticsearch-hadoop/245580 "2020-08-19T10:09:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![HadoopMarc](https://avatars.discourse-cdn.com/v4/letter/h/bbce88/32.png) [@HadoopMarc](https://discuss.elastic.co/u/HadoopMarc)\
**Post date:** [August 19, 2020, 10:09am UTC](https://discuss.elastic.co/t/inner-hits-with-elasticsearch-hadoop/245580/1 "2020-08-19T10:09:56Z")

</div>

Trying here to use a nested query with inner\_hits in combination with elasticsearch-hadoop. I use es-6.8.1 via python using:  
` df = spark.read.format("org.elasticsearch.spark.sql")`

I specify the "es.read.metadata": "true" (otherwise no inner\_hits shows up in the dataframe), resulting in the stacktrace below. Is there anyway to get inner\_hits into the dataframe?

```auto
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, eladsu7061.hadoop.internal, executor 7): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: expected end of object, found END_ARRAY

            at org.elasticsearch.hadoop.util.Assert.isTrue(Assert.java:60)

            at org.elasticsearch.hadoop.serialization.ScrollReader.readHit(ScrollReader.java:433)

            at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:292)

            at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:262)

            at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:313)

            at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:93)

            at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:61)

            at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)

            at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

            at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)

            at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)

            at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)

            at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)

            at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)

            at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)

            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)

            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)

            at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)

            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

            at org.apache.spark.scheduler.Task.run(Task.scala:99)

            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

            at java.lang.Thread.run(Thread.java:748)

```

---

<div class="post-metadata">

**Author:** ![HadoopMarc](https://avatars.discourse-cdn.com/v4/letter/h/bbce88/32.png) [@HadoopMarc](https://discuss.elastic.co/u/HadoopMarc)\
**Post date:** [August 20, 2020, 7:44am UTC](https://discuss.elastic.co/t/inner-hits-with-elasticsearch-hadoop/245580/2 "2020-08-20T07:44:48Z")

</div>

Answer to myself: see related discussion in [https://github.com/elastic/elasticsearch-hadoop/issues/460](https://github.com/elastic/elasticsearch-hadoop/issues/460)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 17, 2020, 7:45am UTC](https://discuss.elastic.co/t/inner-hits-with-elasticsearch-hadoop/245580/3 "2020-09-17T07:45:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.