# I Want to remove the duplicate events inside Logstash filter how could I do that? I mention the events below please have a look and suggest **URL:** https://discuss.elastic.co/t/i-want-to-remove-the-duplicate-events-inside-logstash-filter-how-could-i-do-that-i-mention-the-events-below-please-have-a-look-and-suggest/349175 **Category:** Logstash **Created:** [December 12, 2023, 4:39pm UTC](https://discuss.elastic.co/t/i-want-to-remove-the-duplicate-events-inside-logstash-filter-how-could-i-do-that-i-mention-the-events-below-please-have-a-look-and-suggest/349175 "2023-12-12T16:39:37Z") **Posts on this page:** 1 **Showing post:** 2
### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger) #### Post date: [December 12, 2023, 6:06pm UTC](https://discuss.elastic.co/t/i-want-to-remove-the-duplicate-events-inside-logstash-filter-how-could-i-do-that-i-mention-the-events-below-please-have-a-look-and-suggest/349175/2 "2023-12-12T18:06:50Z")
You can use a fingerprint filter with the [concatenate\_all\_fields](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html#plugins-filters-fingerprint-concatenate_all_fields) option set to true. If you are sending events to elasticsearch then use the fingerprint as the document\_id and duplicate events will be overwritten. If you really want to do the de-duplication in logstash (because you are not writing to elasticsearch) then you would need to use a ruby filter that builds a cache of recently seen fingerprints. You would look for the fingerprint in the cache and event.cancel if it is found, or add it to the cache if not. If you have multiple worker threads then you will need to synchronize access to the cache, and you will need to implement a cache purge strategy. Decidely non-trivial. --- _[View the full topic](https://discuss.elastic.co/t/i-want-to-remove-the-duplicate-events-inside-logstash-filter-how-could-i-do-that-i-mention-the-events-below-please-have-a-look-and-suggest/349175)._