# Mapping es.mapping.id **URL:** \ **Category:** Elasticsearch\ **Created:** [May 12, 2021, 1:38am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755 "2021-05-12T01:38:19Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Gokul6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gokul6/32/84049_2.png) [@Gokul6](https://discuss.elastic.co/u/Gokul6)\ **Post date:** [May 12, 2021, 1:38am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/1 "2021-05-12T01:38:19Z")
i am reading the data from an elasticsearch index, processing it using spark and finally loading it into a new elasticsearch index. while processing the data i am grouping the data using date(day wise) and driverPkid columns present in my data. finally when loading the data into elasticsearch if i use the option ("es.mapping.id" -\> "driverPkid)" only the latest record from my data gets loaded into elasticsearch and the older ones are ignored. when i don't use the option es.mapping.id duplicate records are getting created for the same driverPkid. how to solve this issue.please help. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 12, 2021, 2:24am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/2 "2021-05-12T02:24:53Z")
> [@Gokul6](#): > > es.mapping.id I don't know spark, but it sounds like that assigns the `driverPkid` value from your data to the Elasticsearch `_id`, is that right? And does that value remains the same across different records? If so what you are seeing is valid, as Elasticsearch will be updating the records based on the ID. ---
**Author:** ![Gokul6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gokul6/32/84049_2.png) [@Gokul6](https://discuss.elastic.co/u/Gokul6)\ **Post date:** [May 12, 2021, 2:41am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/3 "2021-05-12T02:41:27Z")
yes!! it assigns the `driverPkid` value from your data to the Elasticsearch `_id`. but the problem is i am grouping the data day wise, so only the latest(today's) record for driverPkid gets loaded in to elasticsearch and the older records (yesterday and day before) for the driverPkid gets ignored. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 12, 2021, 2:42am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/4 "2021-05-12T02:42:18Z")
Does that value stay the same over each day? ---
**Author:** ![Gokul6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gokul6/32/84049_2.png) [@Gokul6](https://discuss.elastic.co/u/Gokul6)\ **Post date:** [May 12, 2021, 2:45am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/5 "2021-05-12T02:45:24Z")
no the value differs. i am calculating the total no of finished orders by each driver per day. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 12, 2021, 2:46am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/6 "2021-05-12T02:46:41Z")
Can you provide some examples of what the documents that you are sending to Elasticsearch look like? ---
**Author:** ![Gokul6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gokul6/32/84049_2.png) [@Gokul6](https://discuss.elastic.co/u/Gokul6)\ **Post date:** [May 12, 2021, 2:57am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/7 "2021-05-12T02:57:57Z")
![data](https://us1.discourse-cdn.com/elastic/original/3X/7/2/721822ef4f2dc8126c6c7d31f93e76532a3a5c2f.png) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 12, 2021, 2:59am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/8 "2021-05-12T02:59:22Z")
The actual json would be handy. ---
**Author:** ![Gokul6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gokul6/32/84049_2.png) [@Gokul6](https://discuss.elastic.co/u/Gokul6)\ **Post date:** [May 12, 2021, 3:01am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/9 "2021-05-12T03:01:55Z")
{ "\_index": "driver-order-details-test", "\_type": "\_doc", "\_id": "daF\_XnkBsBfFsV53dHC2", "\_version": 1, "\_score": null, "\_source": { "driverPkid": 200528, "driverType": "FREELANCER", "totalFIN": 1, "totalCAN": 1, "totalREJ": 0, "the\_date": "2021-05-11" }, "fields": { "the\_date": [ "2021-05-11T00:00:00.000Z" ] }, "sort": [ 1620691200000 ] } ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 12, 2021, 3:05am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/10 "2021-05-12T03:05:08Z")
> [@Gokul6](#): > > "\_id": "daF\_XnkBsBfFsV53dHC2", Ok that looks like an autogenerated ID, and it's not putting `driverPkid` into there as it has it's own field. In that case, I am not sure what is happening, I don't know spark so may not be much help sorry. ---
**Author:** ![Gokul6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gokul6/32/84049_2.png) [@Gokul6](https://discuss.elastic.co/u/Gokul6)\ **Post date:** [May 12, 2021, 3:10am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/11 "2021-05-12T03:10:42Z")
thank you for your time and help @warkolm 🙌 😊 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 9, 2021, 3:11am UTC](https://discuss.elastic.co/t/mapping-es-mapping-id/272755/12 "2021-06-09T03:11:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.