# Replicating Logstash Fingerprint in elasticsearch\_dsl\_py

**URL:** <https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905>\
**Category:** Logstash\
**Created:** [April 21, 2021, 9:39pm UTC](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905 "2021-04-21T21:39:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mark\_Parrish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_parrish/32/58985_2.png) [@Mark\_Parrish](https://discuss.elastic.co/u/Mark_Parrish)\
**Post date:** [April 21, 2021, 9:39pm UTC](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905/1 "2021-04-21T21:39:07Z")

</div>

I am trying replicate this piece of code in the python.

```
ruby {
    code => '
        physical = [
        event.get("address1").to_s,
        event.get("address2").to_s,
        event.get("city").to_s,
        event.get("zip_code").to_s,
        ].join(" ")
        event.set("whole_address", physical.gsub(/[^0-9a-z]/i, "").squeeze(" "))
        event.set("fingerprint", physical.gsub(/[^0-9a-z]/i, "").squeeze(" ").upcase)
    '
}

fingerprint {
    key => "1234ABCD"
    method => "SHA256"
    source => ["fingerprint"]
    target => "[@metadata][generated_id]"
}

```

Using the hashlib module I think it is

[Python Docs](https://docs.python.org/3.8/library/hashlib.html)

```
import hashlib
def add_physical(address1, address2, city, state, zip_code, county):
    p = Place()
    p.address1 = address1
    p.address2 = address2
    p.city = city
    p.state = state
    p.zip_code = zip_code
    p.county = county
    p.whole_address = whole_address(address1, address2, city, zip_code)
    p.type = 'Physical'
    p.geostatus = 'Need'
    p.location = {
        "lon": 0,
        "lat": 0
    }
    key = '1234ABCD'
    fingerprint = whole_address(address1, address2, city, zip_code).upper()
    p.meta.id = hashlib.sha256(fingerprint.encode() + key.encode()).hexdigest()
    p.save()
    Place._index.refresh()
    return p.meta.id

```

But it is not returning the right hash to match the one generated by logstash.

[Logstash Fingerprint Filter](https://github.com/logstash-plugins/logstash-filter-fingerprint/blob/master/lib/logstash/filters/fingerprint.rb)

Any ideas how to make the two match? It helps to prevent duplication.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 21, 2021, 10:14pm UTC](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905/2 "2021-04-21T22:14:48Z")

</div>

You need to use the `hmac` lib in python to create a HMAC hash, as the `fingerprint` in logstash also creates a [HMAC hash](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html#plugins-filters-fingerprint-method).

See this example

```auto
>>> import hashlib
>>> import hmac
>>> salt = '1234ABCD'
>>> message = 'create fingerprint in python'
>>> fingerprint = hmac.new(bytes(salt , 'utf-8'), msg = bytes(message , 'utf-8'), digestmod = hashlib.sha256).hexdigest()
>>> fingerprint
'1ded96f67ddbaa8c586994557d05f1765bb02bb53edaab976cd60402bead8d0d'
>>> 

```

Using the same example in logstash

```auto
fingerprint {
    key => "1234ABCD"
    method => "SHA256"
    source => ["message"]
}

```

The output will be something like this:

```auto
     "@timestamp" => 2021-04-21T22:07:06.991Z,
           "host" => "elk",
        "message" => "create fingerprint in python",
    "fingerprint" => "1ded96f67ddbaa8c586994557d05f1765bb02bb53edaab976cd60402bead8d0d",
       "@version" => "1"
}

```

As you can see the fingerprint from logstash and python are the same.

logstash: `1ded96f67ddbaa8c586994557d05f1765bb02bb53edaab976cd60402bead8d0d`  
python: `1ded96f67ddbaa8c586994557d05f1765bb02bb53edaab976cd60402bead8d0d`

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 21, 2021, 10:25pm UTC](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905/3 "2021-04-21T22:25:49Z")

</div>

Why use a key?

---

<div class="post-metadata">

**Author:** ![Mark\_Parrish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_parrish/32/58985_2.png) [@Mark\_Parrish](https://discuss.elastic.co/u/Mark_Parrish)\
**Post date:** [April 21, 2021, 10:27pm UTC](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905/4 "2021-04-21T22:27:44Z")

</div>

> [@leandrojmp](#):
>
> `import hmac`

Now I see the line in the filter

```
    OpenSSL::HMAC.hexdigest(digest, @key, data.to_s).force_encoding(Encoding::UTF_8)

```

Still not matching. Probably due to how I am formatting the fingerprint. But this is a great start.

---

<div class="post-metadata">

**Author:** ![Mark\_Parrish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_parrish/32/58985_2.png) [@Mark\_Parrish](https://discuss.elastic.co/u/Mark_Parrish)\
**Post date:** [April 21, 2021, 10:29pm UTC](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905/5 "2021-04-21T22:29:06Z")

</div>

I was just copying the code given in the blog.

[Remove Duplication](https://www.elastic.co/blog/how-to-find-and-remove-duplicate-documents-in-elasticsearch)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 21, 2021, 10:46pm UTC](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905/6 "2021-04-21T22:46:25Z")

</div>

That would explain it. That blog is a follow up to another blog, which was published back when using a key (and therefore a MAC rather than a hash) was mandatory. The option to use a simple hash was added in 2018.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 19, 2021, 10:47pm UTC](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905/7 "2021-05-19T22:47:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.