Fingerprint no consistent on combination of same type of value

kishorkumar · March 23, 2024, 2:54pm

Hello Everyone , i am trying to create a unique fingerprint figure print according to upc + displayName and upc + selectedModifiersCombined with the conditions
Here are 4 orders after splitting in logstash
{
orderId: "1548"
items:{
"upc": "1236",
"displayName": "Coffee"
"selectedModifiersCombined": "Coffee Large Hot"}
}
{
orderId: "1648"
items:{
"upc": "1236",
"displayName": "Coffee",
"selectedModifiersCombined": "Coffee Large Hot"}
}{
orderId: "16468"
items:{
"upc": "12386",
"displayName": "Chicken Curry"
}
}{
orderId: "17468"
items:{
"upc": "12386",
"displayName": "Chicken Curry"
}
}I have used this code in order to create this but its create different fingerprints
if [items][selectedModifiers] and [items][selectedModifiers][0] {
fingerprint {
source => ["[items][upc]", "[items][selectedModifiersCombined]"]
target => "fingerprint"
method => "SHA1"
concatenate_all_fields => "true"
}
} else {
fingerprint {
source => ["[items][upc]", "[items][displayName]"]
target => "fingerprint"
method => "SHA1"
concatenate_all_fields => "true"
}
}So why it's not generating same fingerprint for same type of itemshere is the one of the order can you guys help me with that #stack-logstash [#stack-elasticsearch]
(https://elasticstack.slack.com/archives/CNTP6NLTZ)

i have done it with ruby to generate same has and it is working but it does not make sense that why fingerprint is not doing

THis is the first data recorded Order in real test data

"items":[{"name":"657227000506","displayName":"Essentia Purified Water, Ionized Hydration 50.7 fl oz (1 qt 1 pt 2.7 fl oz) 1.5 l","qbName":"","type":"Beverages","subType":"Beverages","upc":"657227000506",

this is the second order in same test data

"items":[{"name":"657227000506","displayName":"Essentia Purified Water, Ionized Hydration 50.7 fl oz (1 qt 1 pt 2.7 fl oz) 1.5 l","qbName":"","type":"Beverages","subType":"Beverages","upc":"657227000506",

Here is the code

if [items][selectedModifiersCombined] {
fingerprint {
source => ["[items][upc]","[items][selectedModifiersCombined]"]
target => "fingerprint"
method => "SHA1"
concatenate_all_fields => true
}
} else {
fingerprint {
source => ["[items][upc]","[items][displayName]"]
target => "fingerprint"
method => "SHA1"
concatenate_all_fields => true
}
}

Here is the output of that using that code

now i here is the ruby code that i have used it's working but i wonder why logstash fingerprint not working

Logstash Elasticsearch

leandrojmp · March 23, 2024, 4:05pm

Hello,

Can you please format your pipeline using the preformatted text option? The </> button. Without it is hard to understand.

Also, where this id field comes from? You highlighted it on your screenshots, but it is not clear from where this id field com from and where this is used.

The values you highlighted are different.

Can you share your entire configuration to make it clear what it is doing? And also some sample messages so this can be replicated.

kishorkumar · March 23, 2024, 5:44pm

the highlighted id just to check that it's different order.

its not used in the fingerprint
.

kishorkumar · March 23, 2024, 5:49pm

leandrojmp · March 23, 2024, 5:57pm

You need to share some samples messages so this can be replicated.

Share a complete sample message where the fingerprint is working and one where it is not working.

kishorkumar · March 23, 2024, 6:41pm

{"userId":"33a4-1ff9-d8-a7fb-d1257","id":"48-0d-44e7-bc-b11","key":"5E74-1218-4F64-B7-947/+1:16:10.502Z/4811","items":[{"name":"657227000506","displayName":"Essentia Purified Water, Ionized Hydration 50.7 fl oz (1 qt 1 pt 2.7 fl oz) 1.5 l","qbName":"","type":"Beverages","subType":"Beverages","upc":"657227000506","modifiers":[],"hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-43515","displayName":"Asian Chicken Salad","qbName":"","type":"The Plant Butcher","subType":"Grab & Go Salads","upc":"43515","selectedModifiers":[{"name":"doms-add-utensil-option-yes","displayName":"Yes","price":0,"upc":"0002","modifierDisplayName":"Utensils","modifierIndex":0,"modifierType":"modifier"}],"hasModifiers":true},{"name":"04904500","displayName":"Coca Cola Diet Coke 20 oz","qbName":"","type":"Beverages","subType":"Beverages","upc":"04904500","hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-12474","displayName":"Fudge Brownie","qbName":"","type":"Desserts","subType":"Dessert Bars","upc":"12474","hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-43502","displayName":"Buffalo Chicken Salad","qbName":"","type":"The Plant Butcher","subType":"Made to Order Salads","upc":"43502","hasModifiers":true,"selectedModifiers":[{"name":"doms-add-utensil-option-yes","displayName":"Yes","price":0,"upc":"0002","modifierDisplayName":"Utensils","modifierIndex":0,"modifierType":"modifier"},{"name":"Dressingontheside-mod-option-0-1699898403266","displayName":"Dressing on the side","price":0,"upc":"MTO1554","modifierDisplayName":"Dressing on the side","modifierIndex":1,"modifierType":"modifier"},{"name":"NoOnion-mod-option-0-1700690395038","displayName":"No Onion","price":0,"upc":"MTO1550","modifierDisplayName":"Remove Ingredients","modifierIndex":3,"modifierType":"modifier"},{"name":"NoCherryTomatoes-mod-option-5-1700690395038","displayName":"No Cherry Tomatoes","price":0,"upc":"MTO1510","modifierDisplayName":"Remove Ingredients","modifierIndex":3,"modifierType":"product"}]}]}
{"userId":"3a4-1ff9-d8-a7fb-13w257","id":"418-0d-44e7-bc-b211","key":"5wE74-1218-4F64-B7-947/+1:16:10.502Z/4811","items":[{"name":"657227000506","displayName":"Essentia Purified Water, Ionized Hydration 50.7 fl oz (1 qt 1 pt 2.7 fl oz) 1.5 l","qbName":"","type":"Beverages","subType":"Beverages","upc":"657227000506","modifiers":[],"hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-43515","displayName":"Asian Chicken Salad","qbName":"","type":"The Plant Butcher","subType":"Grab & Go Salads","upc":"43515","selectedModifiers":[{"name":"doms-add-utensil-option-yes","displayName":"Yes","price":0,"upc":"0002","modifierDisplayName":"Utensils","modifierIndex":0,"modifierType":"modifier"}],"hasModifiers":true},{"name":"04904500","displayName":"Coca Cola Diet Coke 20 oz","qbName":"","type":"Beverages","subType":"Beverages","upc":"04904500","hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-12474","displayName":"Fudge Brownie","qbName":"","type":"Desserts","subType":"Dessert Bars","upc":"12474","hasModifiers":false,"selectedModifiers":[]},{"name":"doms-cst-upc-43502","displayName":"Buffalo Chicken Salad","qbName":"","type":"The Plant Butcher","subType":"Made to Order Salads","upc":"43502","hasModifiers":true,"selectedModifiers":[{"name":"doms-add-utensil-option-yes","displayName":"Yes","price":0,"upc":"0002","modifierDisplayName":"Utensils","modifierIndex":0,"modifierType":"modifier"},{"name":"Dressingontheside-mod-option-0-1699898403266","displayName":"Dressing on the side","price":0,"upc":"MTO1554","modifierDisplayName":"Dressing on the side","modifierIndex":1,"modifierType":"modifier"},{"name":"NoOnion-mod-option-0-1700690395038","displayName":"No Onion","price":0,"upc":"MTO1550","modifierDisplayName":"Remove Ingredients","modifierIndex":3,"modifierType":"modifier"},{"name":"NoCherryTomatoes-mod-option-5-1700690395038","displayName":"No Cherry Tomatoes","price":0,"upc":"MTO1510","modifierDisplayName":"Remove Ingredients","modifierIndex":3,"modifierType":"product"}]}]}

ndjson file

leandrojmp · March 23, 2024, 6:59pm

You didn't share this, it is not possible to know what can be your issue if you do not share how you are processing your logs.

For example, your logs have arrays, which fields are you splitting as individual events and which are you not splitting?

kishorkumar · March 23, 2024, 7:03pm

kishorkumar:

filter {
  json{
    source => "message"
  }

  # Removing extra fields
  mutate {
    remove_field => ["message", "event", "@version", "host", "path"]

  }


  # Splitting the items
  if [items] and [items][0]{
    split {
      field => "[items]"
    }
  }
 


      fingerprint {
          source => ["[items][upc]","[items][displayName]"]
          target => "fingerprint"
          method => "SHA1"
          concatenate_all_fields => true
     }
  

}

the file of filter is give below i think you know how the input works

kishorkumar · March 23, 2024, 7:12pm

i have simplified the work more so don't confuse in if else

leandrojmp · March 23, 2024, 7:14pm

You have a wrong setting:

concatenate_all_fields => true

Using concatenate_all_fields will make logstash concatenate all fields in the event before doing the fingerprint, and since you have different fields, the fingerprint will be different.

What you need to use is concatenate_sources, this will make logstash concatenate only the fields that are specified in the source option, in this case only [items][upc] and [items][displayName].

Replace concatenate_all_fields => true with concatenate_sources => true and it should work, as it worked for me.

With concatenate_sources set as true

$ grep "Essentia Purified" output-discuss.log  | jq .fingerprint
"278fc6caf37c4b923a60741f4f16b37816c91dc2"
"278fc6caf37c4b923a60741f4f16b37816c91dc2"

And with concatenate_all_fields set as true.

$ grep "Essentia Purified" output-discuss.log  | jq .fingerprint
"de543b5ec672b4efa9185a8b53ba5345677137db"
"daefbc9d686489e101f397fb599f13de68dcd615"

kishorkumar · March 23, 2024, 7:23pm

Thanks mate for clarifying the working of concatenate_sources and concatenate_all_fields

system · April 20, 2024, 7:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.