# Found duplicate column(s) in the data schema, Need help on how to load such index data into Spark Dataframe

**URL:** <https://discuss.elastic.co/t/found-duplicate-column-s-in-the-data-schema-need-help-on-how-to-load-such-index-data-into-spark-dataframe/167517>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [February 7, 2019, 7:25pm UTC](https://discuss.elastic.co/t/found-duplicate-column-s-in-the-data-schema-need-help-on-how-to-load-such-index-data-into-spark-dataframe/167517 "2019-02-07T19:25:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yasmeenc](https://avatars.discourse-cdn.com/v4/letter/y/a88e4f/32.png) [@Yasmeenc](https://discuss.elastic.co/u/Yasmeenc)\
**Post date:** [February 7, 2019, 7:25pm UTC](https://discuss.elastic.co/t/found-duplicate-column-s-in-the-data-schema-need-help-on-how-to-load-such-index-data-into-spark-dataframe/167517/1 "2019-02-07T19:25:31Z")

</div>

Hi Team,

I am trying to read data from elasticsearch index and write into a spark dataframe, but the index has same field name with different cases(upper/lower case)

below is the mapping, and the error I am getting is pyspark.sql.utils.AnalysisException: **u'Found duplicate column(s) in the data schema: `providercolumn`;'**

Can you please help on how do I deal with this scenario

{  
“INDEXNAME”: {  
"mappings": {  
“Type”: {  
"properties": {  
**"Providercolumn”: {**  
"type": "text",  
"fields": {  
"keyword": {  
"type": "keyword",  
"ignore\_above": 256  
}  
}  
},  
**"providercolumn”: {**  
"type": "text",  
"fields": {  
"keyword": {  
"type": "keyword",  
"ignore\_above": 256  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [February 11, 2019, 9:50pm UTC](https://discuss.elastic.co/t/found-duplicate-column-s-in-the-data-schema-need-help-on-how-to-load-such-index-data-into-spark-dataframe/167517/2 "2019-02-11T21:50:54Z")

</div>

It seems that Spark is not case sensitive when determining field names. It's most likely a good idea to change the names of these columns if possible, or perhaps even direct Spark to ignore one of them in order for it to successfully read: `es.read.field.exclude = providercolumn`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 11, 2019, 9:50pm UTC](https://discuss.elastic.co/t/found-duplicate-column-s-in-the-data-schema-need-help-on-how-to-load-such-index-data-into-spark-dataframe/167517/3 "2019-03-11T21:50:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.