# How not to duplicate data in Elasticserch

**URL:** <https://discuss.elastic.co/t/how-not-to-duplicate-data-in-elasticserch/305428>\
**Category:** Logstash\
**Tags:** ingest-pipeline\
**Created:** [May 23, 2022, 7:21pm UTC](https://discuss.elastic.co/t/how-not-to-duplicate-data-in-elasticserch/305428 "2022-05-23T19:21:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Daniel\_Oliveira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_oliveira/32/106056_2.png) [@Daniel\_Oliveira](https://discuss.elastic.co/u/Daniel_Oliveira)\
**Post date:** [May 23, 2022, 7:21pm UTC](https://discuss.elastic.co/t/how-not-to-duplicate-data-in-elasticserch/305428/1 "2022-05-23T19:21:03Z")

</div>

Hello gentlemen.

I have a logstash pipeline that processes data from a .csv file

this file has a column called id

how do i not duplicate the data with the same id in case i reprocessed the file or another file that contains the same id?

follow my current filter

```auto
filter {
csv {
separator => ","
skip_header => "true"
columns => ["id","product","job description","uuid Value"]
}

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [May 23, 2022, 7:45pm UTC](https://discuss.elastic.co/t/how-not-to-duplicate-data-in-elasticserch/305428/2 "2022-05-23T19:45:03Z")

</div>

You should use the `document_id` option in the [elasticsearch output](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id).

Something like this:

```auto
output {
    elasticsearch {
        hosts => ["hosts"]
        index => "index-name"
        document_id => "%{id}"
    }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 20, 2022, 7:45pm UTC](https://discuss.elastic.co/t/how-not-to-duplicate-data-in-elasticserch/305428/3 "2022-06-20T19:45:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.