# Improving fingerprint filter performance

**URL:** <https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573>\
**Category:** Logstash\
**Created:** [April 19, 2021, 11:22am UTC](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573 "2021-04-19T11:22:06Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vasu01](https://avatars.discourse-cdn.com/v4/letter/v/c37758/32.png) [@vasu01](https://discuss.elastic.co/u/vasu01)\
**Post date:** [April 19, 2021, 11:22am UTC](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573/1 "2021-04-19T11:22:06Z")

</div>

We are using the logstash fingerprint filter to avoid duplicate data in elasticsearch. But the data ingestion is taking more time.

For Example - A file having ~15000-20000 rows takes approx 2~3 hours to load. Only two fields were given in the source section of the filter.

Here is my configuration:

```auto
fingerprint {
             key => "FINGERPRINT"
             method => "MD5"
             source => ["FILED_1","FIELD_2"]
             target => "document_hash_value"
             concatenate_sources => true
}

```

Is there a way to reduce the ingestion time? Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2021, 3:00pm UTC](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573/2 "2021-04-19T15:00:01Z")

</div>

If you are only able to ingest around two events per second I very much doubt that the problem is in logstash. Try changing the output from elasticsearch to stdout or dots and see what throughput you get then. If it is much higher then the problem is not in the fingerprint filter.

MD5 was deprecated 25 years ago. I would suggest you change that to SHA256 (not SHA1 which has been deprecated for 10 to 15 years, depending on whose recommendations you follow).

---

<div class="post-metadata">

**Author:** ![vasu01](https://avatars.discourse-cdn.com/v4/letter/v/c37758/32.png) [@vasu01](https://discuss.elastic.co/u/vasu01)\
**Post date:** [April 20, 2021, 11:08am UTC](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573/3 "2021-04-20T11:08:49Z")

</div>

Thanks for the inputs. I will try these out!

---

<div class="post-metadata">

**Author:** ![vasu01](https://avatars.discourse-cdn.com/v4/letter/v/c37758/32.png) [@vasu01](https://discuss.elastic.co/u/vasu01)\
**Post date:** [April 26, 2021, 12:21pm UTC](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573/4 "2021-04-26T12:21:44Z")

</div>

@Badger Looks like the issue is not with the fingerprint filter. We have an elasticsearch filter plugin defined which queries es for every record and add a few fields. The fingerprint filter is quick and not the culprit.

Now, Is there any way we can increase the performance of the es filter plugin? I couldn't see any performance-related attributes in the docs.

Also, I noticed that the documents are getting ingested in 250 events per batch. What attribute would increase this setting?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 26, 2021, 5:12pm UTC](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573/5 "2021-04-26T17:12:23Z")

</div>

pipeline.batch.size is set to 125 by default.

What is the network latency between logstash and elasticsearch?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 24, 2021, 5:12pm UTC](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573/6 "2021-05-24T17:12:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.