# SparkSQL Index Mapping, Partition issues

**URL:** <https://discuss.elastic.co/t/sparksql-index-mapping-partition-issues/33419>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [October 30, 2015, 9:03pm UTC](https://discuss.elastic.co/t/sparksql-index-mapping-partition-issues/33419 "2015-10-30T21:03:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tridib](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tridib/32/4727_2.png) [@tridib](https://discuss.elastic.co/u/tridib)\
**Post date:** [October 30, 2015, 9:03pm UTC](https://discuss.elastic.co/t/sparksql-index-mapping-partition-issues/33419/1 "2015-10-30T21:03:36Z")

</div>

I am saving a DataFrame into ElasticSearch. The resource look like as follows: {partitionField}/{docType}  
JavaEsSparkSQL.saveToEs(dataFrame, "{partitionField}/{docType}")

It creates one index for each partition value. There are few issues I am facing:

1. if the partitonField has upper case value elastic search throws exception saying index name can't have upper case.
2. there are nested documents which needs to be mapped as "nested" type. because the Index is created at runtime based on data I am unable to create it before hand with proper mapping. Is there a way to pass the Index specification while saving data to Elastic search?

Thanks & Regards  
Tridib

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [November 7, 2015, 9:16pm UTC](https://discuss.elastic.co/t/sparksql-index-mapping-partition-issues/33419/2 "2015-11-07T21:16:05Z")

</div>

> 1. if the partitonField has upper case value Elasticsearch throws exception saying index name can't have upper case.

That's because Elasticsearch itself does _not_ allow upper case indices. So ES-Hadoop validates the input before deploying the job (only to have it abort) and thus save time.

> Is there a way to pass the Index specification while saving data to Elastic search?

No. This is done on purpose since there's no reliable way across all libraries to load the mapping and save it to Elasticsearch. Also, in practice it would require a JSON mapping and since it's a one time step, it provides no real benefit over the doing things manually especially when using templates (which achieve the same thing directly in ES) as explained [here](https://www.elastic.co/guide/en/elasticsearch/hadoop/master/mapping.html#explicit-mapping).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:27pm UTC](https://discuss.elastic.co/t/sparksql-index-mapping-partition-issues/33419/3 "2017-07-06T13:27:18Z")

</div>