# WARN ScalaRowValueReader: Field 'hits.hits.\_score' is backed by an array but the associated Spark Schema does not reflect this

**URL:** <https://discuss.elastic.co/t/warn-scalarowvaluereader-field-hits-hits-score-is-backed-by-an-array-but-the-associated-spark-schema-does-not-reflect-this/150248>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [September 27, 2018, 7:55pm UTC](https://discuss.elastic.co/t/warn-scalarowvaluereader-field-hits-hits-score-is-backed-by-an-array-but-the-associated-spark-schema-does-not-reflect-this/150248 "2018-09-27T19:55:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Madhur\_Chopra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madhur_chopra/32/33084_2.png) [@Madhur\_Chopra](https://discuss.elastic.co/u/Madhur_Chopra)\
**Post date:** [September 27, 2018, 7:55pm UTC](https://discuss.elastic.co/t/warn-scalarowvaluereader-field-hits-hits-score-is-backed-by-an-array-but-the-associated-spark-schema-does-not-reflect-this/150248/1 "2018-09-27T19:55:09Z")

</div>

i am trying to access \_id field from metadata

```
@udf('string')
def get_id(metadata):
    return metadata['_id']

def get_index(server,index_name,query):
    df = sqlContext.read.format("org.elasticsearch.spark.sql")\
            .option("es.nodes",server)\
            .option("es.query",query)\
            .option("es.read.metadata","true")\
            .option("es.read.metadata.field","metadata")\
            .option("es.read.field.empty.as.null","yes")\
            .option("es.index.read.missing.as.empty","yes")\
            .option("es.read.field.validate.presence","ignore")\
            .load(index_name)
            
    return df.withColumn('Id',get_id(df.metadata)) 

query = """{"query":{ "match_all": {}}}"""

```

The strange part is when i call "get\_index()" and on returned dataframe, I do .select('Id') i see this warning, but when i select all the fields, i do not see this warning.

i have tried adding es.read.field.as.array.include hits.hits.\_score but the warning doesnt seem to go away.

Thanks in advance for help

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 25, 2018, 7:55pm UTC](https://discuss.elastic.co/t/warn-scalarowvaluereader-field-hits-hits-score-is-backed-by-an-array-but-the-associated-spark-schema-does-not-reflect-this/150248/2 "2018-10-25T19:55:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.