# De-duplicating with MURMUR3 vs SHA256

**URL:** <https://discuss.elastic.co/t/de-duplicating-with-murmur3-vs-sha256/147262>\
**Category:** Logstash\
**Created:** [September 4, 2018, 7:35pm UTC](https://discuss.elastic.co/t/de-duplicating-with-murmur3-vs-sha256/147262 "2018-09-04T19:35:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\
**Post date:** [September 4, 2018, 7:35pm UTC](https://discuss.elastic.co/t/de-duplicating-with-murmur3-vs-sha256/147262/1 "2018-09-04T19:35:25Z")

</div>

Hi, has anyone used the fingerprint plugin with MURMUR3? So far I find it has quite high collision rate. Even with just a few hundred thousands records managed to get 20 collisions.

Testing with sha256 over 2million records and no collisions so far. I'm ok with a some collisions. But not what MURMUR3 produced. Just wondering if this article should be updated: [https://www.elastic.co/blog/logstash-lessons-handling-duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)

In both scenarios I'm using the message and the kafka offset as the fields to hash.

---

<div class="post-metadata">

**Author:** ![Mike.Barretta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike.barretta/32/16688_2.png) [@Mike.Barretta](https://discuss.elastic.co/u/Mike.Barretta)\
**Post date:** [September 17, 2018, 3:13pm UTC](https://discuss.elastic.co/t/de-duplicating-with-murmur3-vs-sha256/147262/2 "2018-09-17T15:13:13Z")

</div>

@javadevmtl

Tangent: Have you measured the performance difference of sha256 vs murmur3 in your use case?

Also, FYI, [this issue seeking 128bit murmur3 support](https://github.com/logstash-plugins/logstash-filter-fingerprint/issues/32) might also be worth following if murmur3 is important for you.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 17, 2018, 3:24pm UTC](https://discuss.elastic.co/t/de-duplicating-with-murmur3-vs-sha256/147262/3 "2018-09-17T15:24:17Z")

</div>

SHA 256 is quite long. Whether this is required will depend on the data volume. It might be worthwhile trying out MD5 or SHA1 as well.

---

<div class="post-metadata">

**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\
**Post date:** [September 17, 2018, 7:24pm UTC](https://discuss.elastic.co/t/de-duplicating-with-murmur3-vs-sha256/147262/4 "2018-09-17T19:24:39Z")

</div>

@Mike.Barretta lol that explains it... I thought the logstash murmur3 was the 128bit version.

As for performance it seems the same to me. I eyeballed the graphs of logstash in kibana and they look pretty close... Im running 3 logstash nodes ingesting off 18 partition topic on kafka.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 15, 2018, 7:24pm UTC](https://discuss.elastic.co/t/de-duplicating-with-murmur3-vs-sha256/147262/5 "2018-10-15T19:24:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.