# How to avoid duplicates before entering the filter plugin?

**URL:** https://discuss.elastic.co/t/how-to-avoid-duplicates-before-entering-the-filter-plugin/234735
**Category:** Logstash
**Created:** [May 28, 2020, 11:58am UTC](https://discuss.elastic.co/t/how-to-avoid-duplicates-before-entering-the-filter-plugin/234735 "2020-05-28T11:58:25Z")
**Posts on this page:** 1
**Showing post:** 3

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 28, 2020, 4:12pm UTC](https://discuss.elastic.co/t/how-to-avoid-duplicates-before-entering-the-filter-plugin/234735/3 "2020-05-28T16:12:34Z")

</div>

> [@Nani\_20](#):
>
> Is there any way to identify the duplicate before the **filter plugin** and avoid it?

There are cases where it can be done upstream. For example, a jdbc input might be configured with 'SELECT DISTINCT' which would eliminate duplicates.

In the filters, if you are writing to elasticsearch you might be able to add an elasticsearch filter to query the existence of a document before processing it. But that is not cheap and may not be an optimization. You would need to benchmark both with and without.

---

_[View the full topic](https://discuss.elastic.co/t/how-to-avoid-duplicates-before-entering-the-filter-plugin/234735)._