# Fingerprint- SHA256 takes a lot more space then MURMUR3

**URL:** <https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574>\
**Category:** Logstash\
**Created:** [March 29, 2020, 6:05pm UTC](https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574 "2020-03-29T18:05:15Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![aurimas](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@aurimas](https://discuss.elastic.co/u/aurimas)\
**Post date:** [March 29, 2020, 6:05pm UTC](https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574/1 "2020-03-29T18:05:16Z")

</div>

Hi,  
I have problem with duplicate documents, so I am using method with Logstash described here:

> **[Little Logstash Lessons: Handling Duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)**
>
> Approaches for de-duplicating data in Elasticsearch using Logstash. We also go into examples of how you can use IDs in Elasticsearch Output.

It seems to do the job as after dedublication I get smaller number of documents. But then I get another problem, then I use SHA256 for hashing index takes double amount of space then original and when I use MURMUR3 it takes a little bit less space, witch is normal less documents -\> less space.  
Mapping is identical, and documents themselves look save apart of a lot longer "\_id" with SHA256.  
I can not use MURMUR3 because I have indexes with more documents when MURMUR3 hashing can generate unique IDs.

```
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open telegraf-firewallconnections-2020.01 SIbOVfRCSfCokaEX1ILBZg 1 0 1237508 0 74.2mb 74.2mb
green open telegraf-firewallconnections-2020.01mur3 N6P5fiM-Sm6UfEQlOu2aVQ 1 1 1188297 436 119.4mb 59.8mb
green open telegraf-firewallconnections-2020.01sha256 3nMtJTKcSnCQZxFRGvcrzQ 1 1 1188470 242 305.4mb 152.7mb

```

So why SHA256 takes so much space?

---

<div class="post-metadata">

**Author:** ![aurimas](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@aurimas](https://discuss.elastic.co/u/aurimas)\
**Post date:** [March 29, 2020, 6:13pm UTC](https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574/2 "2020-03-29T18:13:53Z")

</div>

Mapping:

> ```
> {
> "telegraf-firewallconnections-2020.01sha256": {
> "aliases": {},
> "mappings": {
> "doc": {
> "properties": {
> "@timestamp": {
> "type": "date"
> },
> "@version": {
> "type": "keyword",
> "ignore_above": 512
> },
> "firewallconnections": {
> "properties": {
> "firewallmetric1": {
> "type": "float",
> "index": false
> },
> "firewallmetric2": {
> "type": "float",
> "index": false
> },
> "firewallmetric3": {
> "type": "float",
> "index": false
> },
> "firewallmetric4": {
> "type": "float",
> "index": false
> },
> "firewallmetric5": {
> "type": "float",
> "index": false
> },
> "firewallmetric6": {
> "type": "float",
> "index": false
> }
> }
> },
> "measurement_name": {
> "type": "keyword"
> },
> "tag": {
> "properties": {
> "agent_host": {
> "type": "keyword",
> "ignore_above": 512
> },
> "hostname": {
> "type": "keyword",
> "ignore_above": 512
> },
> "index": {
> "type": "keyword",
> "ignore_above": 512
> },
> "measurement_tag": {
> "type": "keyword",
> "ignore_above": 512
> },
> "platform_tag": {
> "type": "keyword",
> "ignore_above": 512
> }
> }
> }
> }
> }
> },
> "settings": {
> "index": {
> "codec": "best_compression",
> "number_of_shards": "1",
> "provided_name": "telegraf-firewallconnections-2020.01sha256",
> "creation_date": "1585503480392",
> "number_of_replicas": "1",
> "uuid": "3nMtJTKcSnCQZxFRGvcrzQ",
> "version": {
> "created": "6080399"
> }
> }
> }
> }
> }
> 
> ```

Logstash config:

```
input {
  elasticsearch {
        hosts => "https://myelastic:9200"
        password => "password"
        user => "uername"
        index => "telegraf-firewallconnections-2020.01"
  }
}
filter {
    fingerprint {
        key => "1234ABCD"
        method => "SHA256"
        source => ["@timestamp","firewallconnections", "tag"]
        target => "[@metadata][generated_id]"
        concatenate_sources => true
    }
    mutate {
        remove_field => ["@version"]
    }
}
output {
        elasticsearch {
        hosts => "https://myelastic:9200"
        password => "password"
        user => "username"
        index => "telegraf-firewallconnections-2020.01sha256"
        document_id => "%{[@metadata][generated_id]}"
    }
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 29, 2020, 7:49pm UTC](https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574/3 "2020-03-29T19:49:12Z")

</div>

SHA256 generates a very long key, so will take up a lot of space. Have you tried a SHA1 hash with base64encode enabled? This is a good hash that is shorter than SHA256 and base64 encoding shrinks it a bit.

---

<div class="post-metadata">

**Author:** ![aurimas](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@aurimas](https://discuss.elastic.co/u/aurimas)\
**Post date:** [March 30, 2020, 1:57pm UTC](https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574/4 "2020-03-30T13:57:31Z")

</div>

@Christian_Dahlqvist thanks for a tip , I have tried all hashing methods and I can see that MD5 with base64encode enabled takes least amount of space, but still more than original 94.8mb compered to 74.2mb.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 30, 2020, 2:00pm UTC](https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574/5 "2020-03-30T14:00:34Z")

</div>

I would expect it to take up more specs, so that is not surprising. It is the price to pay for avoiding duplicates. Be sure that you forcemerge your indices down to 1 segment and index the same data into them for a fair comparison.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2020, 2:00pm UTC](https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574/6 "2020-04-27T14:00:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.