# Logstash duplication

**URL:** <https://discuss.elastic.co/t/logstash-duplication/335847>\
**Category:** Logstash\
**Created:** [June 13, 2023, 9:25am UTC](https://discuss.elastic.co/t/logstash-duplication/335847 "2023-06-13T09:25:09Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ramiwashere](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@ramiwashere](https://discuss.elastic.co/u/ramiwashere)\
**Post date:** [June 13, 2023, 9:25am UTC](https://discuss.elastic.co/t/logstash-duplication/335847/1 "2023-06-13T09:25:09Z")

</div>

Hello

I have created a logstash pipeline via the http\_poller plugin in order to collect information from an API link.

In order to manage the duplication of documents, I used the 'fingerprint' plugin in the filter part of the logstash pipeline. This seems to work without a hitch, as I always have a single example document in the index.

However, each time the logstash pipeline is run, the documents are updated again to the date they were imported into the index.

Is it possible to ingest only those documents that are not already in the index?  
Below, the piepline elements:

```auto
}

filter {
    mutate {
        rename => {"id" => "cve_id"}
    }
    fingerprint {
         source => "cve_id"
         method => "SHA256"
         target => "[@metadata][fingerprint]"
    }
}

output {
    elasticsearch {
		index => "opencve"
		hosts => ***
		user => ***
		password => ***
        ssl_certificate_verification => false
        document_id => "%{[@metadata][fingerprint]}"
	}
}

```

Thanks for your help

---

<div class="post-metadata">

**Author:** ![Priscilla\_Parodi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priscilla_parodi/32/43047_2.png) [@Priscilla\_Parodi](https://discuss.elastic.co/u/Priscilla_Parodi)\
**Post date:** [June 29, 2023, 2:11pm UTC](https://discuss.elastic.co/t/logstash-duplication/335847/2 "2023-06-29T14:11:27Z")

</div>

Hello @ramiwashere,

If you want to add only new documents to the index, you can use the [action](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action) "create" parameter. It'll fail if a document with the same ID already exists in the index.

```auto
output {
  elasticsearch {
    ...
    action => "create"
  }
}

```

Hope it helps!

---

<div class="post-metadata">

**Author:** ![ramiwashere](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@ramiwashere](https://discuss.elastic.co/u/ramiwashere)\
**Post date:** [June 30, 2023, 3:06pm UTC](https://discuss.elastic.co/t/logstash-duplication/335847/3 "2023-06-30T15:06:12Z")

</div>

Hello Priscilla,

Thank you  
I just update my pipeline and I will let you know if its work !

Regards,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 28, 2023, 3:06pm UTC](https://discuss.elastic.co/t/logstash-duplication/335847/4 "2023-07-28T15:06:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.