# How to remove duplicate values in ealstic search

**URL:** <https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226>\
**Category:** Elasticsearch\
**Created:** [December 18, 2017, 12:16pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226 "2017-12-18T12:16:31Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![narasiman1986](https://avatars.discourse-cdn.com/v4/letter/n/ce73a5/32.png) [@narasiman1986](https://discuss.elastic.co/u/narasiman1986)\
**Post date:** [December 19, 2017, 10:47am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226/21 "2017-12-19T10:47:04Z")

</div>

i format my code using \</\> icon and send to again.

input {

```
jdbc {

jdbc_driver_library => "D:\mysql-connector-java-5.1.44\mysql-connector-java-5.1.44\mysql-connector-java-5.1.44-bin.jar"

jdbc_driver_class => "com.mysql.jdbc.Driver"

 
jdbc_connection_string => "jdbc:mysql://localhost:3306/sample"

  
jdbc_user => "root"

jdbc_password => "root"

jdbc_fetch_size => 10000

    schedule => "* * * * *"
    statement => "SELECT * from sample"

#codec => "json"

  }

}

filter {
  fingerprint {
    source => "RRH_MR_NUM"
    target => "[@metadata][fingerprint]"
    method => "MURMUR3"
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"] 
	index => "clinical" 
    # document_id => "%{[@metadata][fingerprint]}"	
	}
	stdout { codec => rubydebug }
    }

```

i tried this above query to eliminate the duplicate records in elastic search.

is there any mistake in my above code.

please let me know.

---

<div class="post-metadata">

**Author:** ![Arthur\_Silva\_Sens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arthur_silva_sens/32/24913_2.png) [@Arthur\_Silva\_Sens](https://discuss.elastic.co/u/Arthur_Silva_Sens)\
**Post date:** [December 19, 2017, 10:57am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226/22 "2017-12-19T10:57:38Z")

</div>

One strategy that you could use is to query where "mrdno" is "11657", then save **ONE** of the ids

Then delete where "mrdno" **is**"11657" AND \_id NOT = _the id you saved_

It should delete all the duplicates

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 19, 2017, 11:06am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226/23 "2017-12-19T11:06:46Z")

</div>

About the format: please try to indent your code like you did for the `fingerprint` part.

Why this is commented?

```
# document_id => "%{[@metadata][fingerprint]}"
```

---

<div class="post-metadata">

**Author:** ![narasiman1986](https://avatars.discourse-cdn.com/v4/letter/n/ce73a5/32.png) [@narasiman1986](https://discuss.elastic.co/u/narasiman1986)\
**Post date:** [December 19, 2017, 11:27am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226/24 "2017-12-19T11:27:42Z")

</div>

i am not commented the below line.

how you are telling i am commented the below line.

# document\_id =\> "%{[@metadata][fingerprint]}"

you are saying # means comment?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 19, 2017, 11:44am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226/25 "2017-12-19T11:44:38Z")

</div>

Seriously... Please...

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/6/a6ca944fcb21cc2ebd9bfbb3b2d0d59d65929ebd.jpg)

---

<div class="post-metadata">

**Author:** ![abdbaddude](https://avatars.discourse-cdn.com/v4/letter/a/c2a13f/32.png) [@abdbaddude](https://discuss.elastic.co/u/abdbaddude)\
**Post date:** [December 22, 2017, 2:00pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226/26 "2017-12-22T14:00:33Z")

</div>

🤣

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 19, 2018, 2:00pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226/27 "2018-01-19T14:00:57Z")

</div>

Closed on January 19, 2018, 2:00pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![Alex\_Marquardt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_marquardt/32/42925_2.png) [@Alex\_Marquardt](https://discuss.elastic.co/u/Alex_Marquardt)\
**Post date:** [July 26, 2018, 8:00pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226/28 "2018-07-26T20:00:13Z")

</div>

I have written a blog post about removing duplicate documents from Elasticsearch, which can be found at [https://alexmarquardt.com/2018/07/23/deduplicating-documents-in-elasticsearch/](https://alexmarquardt.com/2018/07/23/deduplicating-documents-in-elasticsearch/)

[Previous page](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-ealstic-search/112226.md?page=1)
