# How ES schema is determined while reading using hadoop

**URL:** <https://discuss.elastic.co/t/how-es-schema-is-determined-while-reading-using-hadoop/167694>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [February 9, 2019, 12:11am UTC](https://discuss.elastic.co/t/how-es-schema-is-determined-while-reading-using-hadoop/167694 "2019-02-09T00:11:25Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sonny\_Heer](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Sonny\_Heer](https://discuss.elastic.co/u/Sonny_Heer)\
**Post date:** [February 9, 2019, 12:11am UTC](https://discuss.elastic.co/t/how-es-schema-is-determined-while-reading-using-hadoop/167694/1 "2019-02-09T00:11:25Z")

</div>

Env:  
using this jar: elasticsearch-spark-13\_2.10-5.1.1.jar

`sqlContext.read.format("org.elasticsearch.spark.sql").option("es.nodes",es_url).option("es.port", "443").option("es.nodes.wan.only", "true").option("es.net.ssl", "true").option("es.read.field.as.array.include",array_with_comma).option("es.mapping.date.rich","false").option("es.read.field.exclude", exclude_with_comma).option("es.read.field.include", "").option("pushdown", "true").load(es_index)`

not passing any args except for exclude fields in which case we exclude a couple from top level.

The problem we have is missing fields in the dataframe.printSchema()...

Does it use \_mapping to figure out the schema or sampling? I didn't find any docs on this.

Thanks.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [February 11, 2019, 9:54pm UTC](https://discuss.elastic.co/t/how-es-schema-is-determined-while-reading-using-hadoop/167694/2 "2019-02-11T21:54:08Z")

</div>

ES-Hadoop uses the mapping endpoint for the resource given, though in the 5.x line there is a bug when attempting to read from multiple indices or types: When the schema is discovered at the start of the process, only one mapping is picked up and used for the fields. I would check to make sure that you are only reading from one index and type, or if you are reading from multiple, ensure that their mappings are identical across the board

---

<div class="post-metadata">

**Author:** ![Sonny\_Heer](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Sonny\_Heer](https://discuss.elastic.co/u/Sonny_Heer)\
**Post date:** [February 11, 2019, 11:06pm UTC](https://discuss.elastic.co/t/how-es-schema-is-determined-while-reading-using-hadoop/167694/3 "2019-02-11T23:06:20Z")

</div>

Thanks James! That helps. It appears the mapping isn't being updated when data is added by the other team - causing our issue. thanks again for confirming.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 11, 2019, 11:06pm UTC](https://discuss.elastic.co/t/how-es-schema-is-determined-while-reading-using-hadoop/167694/4 "2019-03-11T23:06:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.