# Avoiding duplicate records

**URL:** <https://discuss.elastic.co/t/avoiding-duplicate-records/102471>\
**Category:** Logstash\
**Created:** [October 2, 2017, 7:05pm UTC](https://discuss.elastic.co/t/avoiding-duplicate-records/102471 "2017-10-02T19:05:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![bgoldowsky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bgoldowsky/32/22664_2.png) [@bgoldowsky](https://discuss.elastic.co/u/bgoldowsky)\
**Post date:** [October 2, 2017, 7:05pm UTC](https://discuss.elastic.co/t/avoiding-duplicate-records/102471/1 "2017-10-02T19:05:16Z")

</div>

I am using the logstash JDBC input plugin to push new rows from a database query into Elasticsearch, updating any old items that have changed. I'd like to avoid duplicates, so I tried to use an "upsert" pattern:

output {  
elasticsearch {  
hosts =\> "http://....:9200"  
index =\> "snudle-qa-event-%{+YYYY.MM.dd}"  
action =\> "update"  
doc\_as\_upsert =\> true  
document\_id =\> "%{id}"  
}  
}

However, I am still seeing duplicates - in addition to the records with my database IDs, there are some additional records with random-looking new Id values, eg: AV7dtU2XQygf4KBYvrIq .

Any clues what I am doing wrong?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 4, 2017, 8:41pm UTC](https://discuss.elastic.co/t/avoiding-duplicate-records/102471/2 "2017-10-04T20:41:21Z")

</div>

Wild guess: Do you have any extra files in /etc/logstash/conf.d (or wherever your configuration files are stored)?

With the configuration you've shown, the document id will always be identical to the contents of the `id` field.

---

<div class="post-metadata">

**Author:** ![bgoldowsky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bgoldowsky/32/22664_2.png) [@bgoldowsky](https://discuss.elastic.co/u/bgoldowsky)\
**Post date:** [October 5, 2017, 2:49pm UTC](https://discuss.elastic.co/t/avoiding-duplicate-records/102471/3 "2017-10-05T14:49:14Z")

</div>

Hm, there's my project.conf file, a project.conf~ and a project.conf.bak representing previous versions. Would those get read? I'm used to apache, which only considers files with the expected .conf suffix.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 5, 2017, 6:40pm UTC](https://discuss.elastic.co/t/avoiding-duplicate-records/102471/4 "2017-10-05T18:40:41Z")

</div>

> Would those get read?

Yes. I think Logstash 6.0 only reads \*.conf though.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 2, 2017, 6:40pm UTC](https://discuss.elastic.co/t/avoiding-duplicate-records/102471/5 "2017-11-02T18:40:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.