# Spark DataFrame -- Elastic Seach write \_ID **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [February 26, 2017, 5:46pm UTC](https://discuss.elastic.co/t/spark-dataframe-elastic-seach-write--id/76551 "2017-02-26T17:46:04Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Muthu\_Jayakumar](https://avatars.discourse-cdn.com/v4/letter/m/49beb7/32.png) [@Muthu\_Jayakumar](https://discuss.elastic.co/u/Muthu_Jayakumar)\ **Post date:** [February 26, 2017, 5:46pm UTC](https://discuss.elastic.co/t/spark-dataframe-elastic-seach-write--id/76551/1 "2017-02-26T17:46:04Z")
Hello there, I am trying see if there is a way to set/override `_id` attribute while writing a dataframe into elastic-search. Not sure if I have a column named "\_id" can help. I would like to set the "\_id" field so that duplicate rows are not inserted into elastic search. Please advice, Muthu ---
**Author:** ![Muthu\_Jayakumar](https://avatars.discourse-cdn.com/v4/letter/m/49beb7/32.png) [@Muthu\_Jayakumar](https://discuss.elastic.co/u/Muthu_Jayakumar)\ **Post date:** [February 26, 2017, 6:02pm UTC](https://discuss.elastic.co/t/spark-dataframe-elastic-seach-write--id/76551/2 "2017-02-26T18:02:59Z")
Not sure if `es.mapping.id` (ref: https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html) would do the trick? But if I have 2 rows with the same id field, would one row overwrite the other? ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [March 11, 2017, 8:05pm UTC](https://discuss.elastic.co/t/spark-dataframe-elastic-seach-write--id/76551/3 "2017-03-11T20:05:56Z")
@Muthu_Jayakumar That is indeed the correct setting for defining an ID on a document by document basis. Make sure to have a unique ID for each document or if the ID's are not unique to de-duplicate and collapse the data per id before ingesting into Elasticsearch. Since Spark and Hadoop tasks are not run in a deterministic order, only the data within a task is correctly ordered, and you may see non deterministic write results if those previous steps are not taken. ---
**Author:** ![Muthu\_Jayakumar](https://avatars.discourse-cdn.com/v4/letter/m/49beb7/32.png) [@Muthu\_Jayakumar](https://discuss.elastic.co/u/Muthu_Jayakumar)\ **Post date:** [March 12, 2017, 7:28pm UTC](https://discuss.elastic.co/t/spark-dataframe-elastic-seach-write--id/76551/4 "2017-03-12T19:28:40Z")
Hello James Baiera, I have taken the route of coming up `_id` attribute because the same `spark dataframe`, when seen from elastic search has duplicate rows. But during all these time, when I see the `dataframe` from spark side of things, it seems about right. Again, this issue happens for the exact same datasource-dataframe in 1 out of 10 times. Other times the data in elastic search looks fine. I am using Elastic Search 5.2.1and Apache Spark 2.1.0. Please advice, Muthu ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [March 12, 2017, 8:10pm UTC](https://discuss.elastic.co/t/spark-dataframe-elastic-seach-write--id/76551/5 "2017-03-12T20:10:57Z")
Generally it is advised to set an id field since task failures can cause duplicate data to be inserted when they are retried. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 9, 2017, 8:11pm UTC](https://discuss.elastic.co/t/spark-dataframe-elastic-seach-write--id/76551/6 "2017-04-09T20:11:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.