# Unclear how logstash fingerprint treats empty/non-existing fields

**URL:** <https://discuss.elastic.co/t/unclear-how-logstash-fingerprint-treats-empty-non-existing-fields/281157>\
**Category:** Logstash\
**Created:** [August 12, 2021, 6:45am UTC](https://discuss.elastic.co/t/unclear-how-logstash-fingerprint-treats-empty-non-existing-fields/281157 "2021-08-12T06:45:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![AdmireDirac](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@AdmireDirac](https://discuss.elastic.co/u/AdmireDirac)\
**Post date:** [August 12, 2021, 6:45am UTC](https://discuss.elastic.co/t/unclear-how-logstash-fingerprint-treats-empty-non-existing-fields/281157/1 "2021-08-12T06:45:08Z")

</div>

Hello everyone,

I have a hard time to understand the behaviour fo the logstash fingerprint filter if a field is empty or non-existent.  
My scenario is the following:

I'm reading a json-file with logstash and create a document for each json object in an elasticsearch index.  
I'm calculating an MD5 fingerprint from some of the key-value-pairs (including nested objects) composing the json object.

```auto
{
    "key_a": "value_a",
    "key_b": "value_b",
    "key_c": "value_c",
    "key_d": {
        "key_e": "value_e",
        "key_f": "value_f"
    }
}

```

Fingerprint filter:

```auto
filter {
    fingerprint {
        source => ["key_a", "key_b", "key_d"]
        concatenate_sources => true
        concatenate_all_fields => true
        target => "doc_fingerprint"
        method => "MD5"
        key => "integrity"
    }
}

```

As you can see from the example above, I am not using all key-value-pairs of the json object.  
Moreover not all keys are present in every json object. For example in some instances the "key\_b" is missing. In other instances "key\_a" can have no value: "" or nil

Now to the actual issue:  
I have a second logstash-file which I use to read all the documents from the previously filled elasticsearch index and to calculate the fingerprint again. The filter remains unchanged (except for the target) since the field name in the documents and the keys in the json object are identical.

```auto
filter {
    fingerprint {
        source => ["key_a", "key_b", "key_d"]
        concatenate_sources => true
        concatenate_all_fields => true
        target => "second_doc_fingerprint"
        method => "MD5"
        key => "integrity"
    }

```

However, this condition following the fingerprint filter is always false:

```auto

    if second_doc_fingerprint != doc_fingerprint {
        ruby { code => 'puts "unequal"' }
    }
}

```

I'm trying to find a debug interface to understand what's happening here.  
Is it possible that the empty or non-exisiting fields are related to the issue?  
I tried to recalculate the fingerprint with a python script unsuccessfully.  
I found this thread  
[https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905)  
but I am not sure how to compose the string which is hashed when the following settings are made:

```auto
        concatenate_sources => true
        concatenate_all_fields => true

```

It would be great if you can help me out 🙂 Thank you in advance!

---

<div class="post-metadata">

**Author:** ![AdmireDirac](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@AdmireDirac](https://discuss.elastic.co/u/AdmireDirac)\
**Post date:** [August 12, 2021, 12:00pm UTC](https://discuss.elastic.co/t/unclear-how-logstash-fingerprint-treats-empty-non-existing-fields/281157/2 "2021-08-12T12:00:19Z")

</div>

I found my error:  
I misunderstood the

```auto
concatenate_all_fields => true

```

flag. Of course this parameter take into account not only the fields given in "source" but also the other fields including the hash which was calculated in the first run.

What I haven't figured out yet is how to compute the hash with a different method like e.g. with python. Especially nested fields give me a hard time.

But this is a different question and this one here can be marked as solved.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 9, 2021, 12:01pm UTC](https://discuss.elastic.co/t/unclear-how-logstash-fingerprint-treats-empty-non-existing-fields/281157/3 "2021-09-09T12:01:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.