# \[elasticsearch-hadoop\] How to specify es.mapping.id value from inside a map?

**URL:** <https://discuss.elastic.co/t/elasticsearch-hadoop-how-to-specify-es-mapping-id-value-from-inside-a-map/112562>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [December 20, 2017, 7:01am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-how-to-specify-es-mapping-id-value-from-inside-a-map/112562 "2017-12-20T07:01:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![akshru](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@akshru](https://discuss.elastic.co/u/akshru)\
**Post date:** [December 20, 2017, 7:01am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-how-to-specify-es-mapping-id-value-from-inside-a-map/112562/1 "2017-12-20T07:01:07Z")

</div>

Hi,

I am working using elasticsearch-hadoop dependency for uploading dataframes to AWS ES service.  
I want to specify \_id for every document that has to be uploaded. However the document id resides inside a map in my document to be uploaded.  
Sample document that I upload to AWS elasticsearch -  
{  
"transactionIds": {  
"purchaseId": "PUID:1234-5678-910"  
},  
"purchaseAttempts": [],  
"timestamp": "2017-12-20T06:39:51.299Z[UTC]",  
"year": 2017,  
"month": 12,  
"day": 20,  
"hour": 6  
}

I want purchaseId to be my document's \_id.  
How can I provide the entry of "purchaseId" present inside map of "transactionIds" to be the field for "[es.mapping.id](http://es.mapping.id)"?

Here is the ES configuration that I have tried but fails-  
"[es.mapping.id](http://es.mapping.id)", "transactionIds.purchaseId"

I get runtime exception when I try to use the above config-

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: [DataFrameFieldExtractor for field [[transactionIds, purchaseId]]] cannot extract value from entity [class java.lang.String] | instance [([[null,null,null,null,PUID:934-5439103-6862003,null],WrappedArray(),2017-12-20T06:59:03.900Z[UTC],2017,12,20,6],StructType(StructField(transactionIds,StructType(StructField(transactionId,StringType,true), StructField(marketplaceId,StringType,true), StructField(customerId,StringType,true), StructField(sessionId,StringType,true), StructField(purchaseId,StringType,true), StructField(orderIds,ArrayType(StringType,true),true)),true), StructField(purchaseAttempts,ArrayType(StructType(StructField(attemptId,StringType,true), StructField(attemptEvents,MapType(StringType,StructType(StructField(eventType,StringType,true), StructField(idType,StringType,true), StructField(id,StringType,true), StructField(startTime,LongType,true), StructField(endTime,LongType,true), StructField(attributes,MapType(StringType,StringType,true),true), StructField(paymentOption,StructType(StructField(paymentMethod,StringType,true), StructField(paymentAttrs,MapType(StringType,StringType,true),true)),true)),true),true)),true),true), StructField(timestamp,StringType,true), StructField(year,IntegerType,true), StructField(month,IntegerType,true), StructField(day,IntegerType,true), StructField(hour,IntegerType,true)))]  
at org.elasticsearch.hadoop.serialization.bulk.AbstractBulkFactory$FieldWriter.write(AbstractBulkFactory.java:98)  
at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.writeTemplate(TemplatedBulk.java:80)  
at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.write(TemplatedBulk.java:56)  
at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:159)  
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:67)  
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:97)  
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:97)  
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)  
at org.apache.spark.scheduler.Task.run(Task.scala:99)  
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:745)

---

<div class="post-metadata">

**Author:** ![akshru](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@akshru](https://discuss.elastic.co/u/akshru)\
**Post date:** [December 20, 2017, 7:03am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-how-to-specify-es-mapping-id-value-from-inside-a-map/112562/2 "2017-12-20T07:03:11Z")

</div>

Specifying a non-map value like "timestamp" in the above document to be uploaded to ES works fine.  
ie.  
"[es.mapping.id](http://es.mapping.id)", "timestamp" =\> works fine  
"[es.mapping.id](http://es.mapping.id)", "transactionIds.purchaseId" =\> throws exception

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2018, 7:03am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-how-to-specify-es-mapping-id-value-from-inside-a-map/112562/3 "2018-01-17T07:03:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.