Fingerprint no consistent on combination of same type of value

Hello Everyone , i am trying to create a unique fingerprint figure print according to upc + displayName and upc + selectedModifiersCombined with the conditions
Here are 4 orders after splitting in logstash
{
orderId: "1548"
items:{
"upc": "1236",
"displayName": "Coffee"
"selectedModifiersCombined": "Coffee Large Hot"}
}
{
orderId: "1648"
items:{
"upc": "1236",
"displayName": "Coffee",
"selectedModifiersCombined": "Coffee Large Hot"}
}{
orderId: "16468"
items:{
"upc": "12386",
"displayName": "Chicken Curry"
}
}{
orderId: "17468"
items:{
"upc": "12386",
"displayName": "Chicken Curry"
}
}I have used this code in order to create this but its create different fingerprints
if [items][selectedModifiers] and [items][selectedModifiers][0] {
fingerprint {
source => ["[items][upc]", "[items][selectedModifiersCombined]"]
target => "fingerprint"
method => "SHA1"
concatenate_all_fields => "true"
}
} else {
fingerprint {
source => ["[items][upc]", "[items][displayName]"]
target => "fingerprint"
method => "SHA1"
concatenate_all_fields => "true"
}
}So why it's not generating same fingerprint for same type of itemshere is the one of the order can you guys help me with that #stack-logstash [#stack-elasticsearch]
(https://elasticstack.slack.com/archives/CNTP6NLTZ)

i have done it with ruby to generate same has and it is working but it does not make sense that why fingerprint is not doing

THis is the first data recorded Order in real test data

"items":[{"name":"657227000506","displayName":"Essentia Purified Water, Ionized Hydration 50.7 fl oz (1 qt 1 pt 2.7 fl oz) 1.5 l","qbName":"","type":"Beverages","subType":"Beverages","upc":"657227000506",

this is the second order in same test data

"items":[{"name":"657227000506","displayName":"Essentia Purified Water, Ionized Hydration 50.7 fl oz (1 qt 1 pt 2.7 fl oz) 1.5 l","qbName":"","type":"Beverages","subType":"Beverages","upc":"657227000506",

Here is the code

if [items][selectedModifiersCombined] {
fingerprint {
source => ["[items][upc]","[items][selectedModifiersCombined]"]
target => "fingerprint"
method => "SHA1"
concatenate_all_fields => true
}
} else {
fingerprint {
source => ["[items][upc]","[items][displayName]"]
target => "fingerprint"
method => "SHA1"
concatenate_all_fields => true
}
}

Here is the output of that using that code

now i here is the ruby code that i have used it's working but i wonder why logstash fingerprint not working

Logstash Elasticsearch

Hello,

Can you please format your pipeline using the preformatted text option? The </> button. Without it is hard to understand.

Also, where this id field comes from? You highlighted it on your screenshots, but it is not clear from where this id field com from and where this is used.

The values you highlighted are different.

Can you share your entire configuration to make it clear what it is doing? And also some sample messages so this can be replicated.

the highlighted id just to check that it's different order.

its not used in the fingerprint
.

image

You need to share some samples messages so this can be replicated.

Share a complete sample message where the fingerprint is working and one where it is not working.

{"userId":"33a4-1ff9-d8-a7fb-d1257","id":"48-0d-44e7-bc-b11","key":"5E74-1218-4F64-B7-947/+1:16:10.502Z/4811","items":[{"name":"657227000506","displayName":"Essentia Purified Water, Ionized Hydration 50.7 fl oz (1 qt 1 pt 2.7 fl oz) 1.5 l","qbName":"","type":"Beverages","subType":"Beverages","upc":"657227000506","modifiers":[],"hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-43515","displayName":"Asian Chicken Salad","qbName":"","type":"The Plant Butcher","subType":"Grab & Go Salads","upc":"43515","selectedModifiers":[{"name":"doms-add-utensil-option-yes","displayName":"Yes","price":0,"upc":"0002","modifierDisplayName":"Utensils","modifierIndex":0,"modifierType":"modifier"}],"hasModifiers":true},{"name":"04904500","displayName":"Coca Cola Diet Coke 20 oz","qbName":"","type":"Beverages","subType":"Beverages","upc":"04904500","hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-12474","displayName":"Fudge Brownie","qbName":"","type":"Desserts","subType":"Dessert Bars","upc":"12474","hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-43502","displayName":"Buffalo Chicken Salad","qbName":"","type":"The Plant Butcher","subType":"Made to Order Salads","upc":"43502","hasModifiers":true,"selectedModifiers":[{"name":"doms-add-utensil-option-yes","displayName":"Yes","price":0,"upc":"0002","modifierDisplayName":"Utensils","modifierIndex":0,"modifierType":"modifier"},{"name":"Dressingontheside-mod-option-0-1699898403266","displayName":"Dressing on the side","price":0,"upc":"MTO1554","modifierDisplayName":"Dressing on the side","modifierIndex":1,"modifierType":"modifier"},{"name":"NoOnion-mod-option-0-1700690395038","displayName":"No Onion","price":0,"upc":"MTO1550","modifierDisplayName":"Remove Ingredients","modifierIndex":3,"modifierType":"modifier"},{"name":"NoCherryTomatoes-mod-option-5-1700690395038","displayName":"No Cherry Tomatoes","price":0,"upc":"MTO1510","modifierDisplayName":"Remove Ingredients","modifierIndex":3,"modifierType":"product"}]}]}
{"userId":"3a4-1ff9-d8-a7fb-13w257","id":"418-0d-44e7-bc-b211","key":"5wE74-1218-4F64-B7-947/+1:16:10.502Z/4811","items":[{"name":"657227000506","displayName":"Essentia Purified Water, Ionized Hydration 50.7 fl oz (1 qt 1 pt 2.7 fl oz) 1.5 l","qbName":"","type":"Beverages","subType":"Beverages","upc":"657227000506","modifiers":[],"hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-43515","displayName":"Asian Chicken Salad","qbName":"","type":"The Plant Butcher","subType":"Grab & Go Salads","upc":"43515","selectedModifiers":[{"name":"doms-add-utensil-option-yes","displayName":"Yes","price":0,"upc":"0002","modifierDisplayName":"Utensils","modifierIndex":0,"modifierType":"modifier"}],"hasModifiers":true},{"name":"04904500","displayName":"Coca Cola Diet Coke 20 oz","qbName":"","type":"Beverages","subType":"Beverages","upc":"04904500","hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-12474","displayName":"Fudge Brownie","qbName":"","type":"Desserts","subType":"Dessert Bars","upc":"12474","hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-43502","displayName":"Buffalo Chicken Salad","qbName":"","type":"The Plant Butcher","subType":"Made to Order Salads","upc":"43502","hasModifiers":true,"selectedModifiers":[{"name":"doms-add-utensil-option-yes","displayName":"Yes","price":0,"upc":"0002","modifierDisplayName":"Utensils","modifierIndex":0,"modifierType":"modifier"},{"name":"Dressingontheside-mod-option-0-1699898403266","displayName":"Dressing on the side","price":0,"upc":"MTO1554","modifierDisplayName":"Dressing on the side","modifierIndex":1,"modifierType":"modifier"},{"name":"NoOnion-mod-option-0-1700690395038","displayName":"No Onion","price":0,"upc":"MTO1550","modifierDisplayName":"Remove Ingredients","modifierIndex":3,"modifierType":"modifier"},{"name":"NoCherryTomatoes-mod-option-5-1700690395038","displayName":"No Cherry Tomatoes","price":0,"upc":"MTO1510","modifierDisplayName":"Remove Ingredients","modifierIndex":3,"modifierType":"product"}]}]}

ndjson file

You didn't share this, it is not possible to know what can be your issue if you do not share how you are processing your logs.

For example, your logs have arrays, which fields are you splitting as individual events and which are you not splitting?

the file of filter is give below i think you know how the input works

i have simplified the work more so don't confuse in if else

You have a wrong setting:

concatenate_all_fields => true

Using concatenate_all_fields will make logstash concatenate all fields in the event before doing the fingerprint, and since you have different fields, the fingerprint will be different.

What you need to use is concatenate_sources, this will make logstash concatenate only the fields that are specified in the source option, in this case only [items][upc] and [items][displayName].

Replace concatenate_all_fields => true with concatenate_sources => true and it should work, as it worked for me.

With concatenate_sources set as true

$ grep "Essentia Purified" output-discuss.log  | jq .fingerprint
"278fc6caf37c4b923a60741f4f16b37816c91dc2"
"278fc6caf37c4b923a60741f4f16b37816c91dc2"

And with concatenate_all_fields set as true.

$ grep "Essentia Purified" output-discuss.log  | jq .fingerprint
"de543b5ec672b4efa9185a8b53ba5345677137db"
"daefbc9d686489e101f397fb599f13de68dcd615"

Thanks mate for clarifying the working of concatenate_sources and concatenate_all_fields

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.