# Logstash fingerprint not able to remove duplicate
**URL:** https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663
**Category:** Logstash
**Created:** [September 23, 2019, 10:13am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663 "2019-09-23T10:13:39Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@sharry007](https://discuss.elastic.co/u/sharry007)
#### Post date: [September 23, 2019, 10:13am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663/1 "2019-09-23T10:13:39Z")
Hi,
We have been using logstash fingerprint plugin since quite sometime. We now have a specific requirement of creating fingerprint hashkey using multiple fields of Elasticsearch indexes, but we are stuck in two cases as detailed below,
If concatenate\_sources = False,
Even if we provide multiple fields, it seems to use only last field to generate hashkey.
If concatenate\_source = True,
If using multiple fields, duplicate data is eliminated when data is consumed from a single file in s3. If similar duplicate data is consumed from multiple S3 files, it fails to eliminate duplicacy.
Please suggest how to successfully eliminate duplicacy while reading data from several files in S3 bucket
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 23, 2019, 12:05pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663/2 "2019-09-23T12:05:57Z")
Which fields are you specifying for the hash?
---
### Author:  [@sharry007](https://discuss.elastic.co/u/sharry007)
#### Post date: [September 24, 2019, 4:03am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663/3 "2019-09-24T04:03:53Z")
Hello Christian,
Please find below sample data and our filter inclusive of fingerprint config
{"logtype": "logs","servicename": "Ref","deviceinfo": {"productid": "ARTIK051","policyversion": 1541130000,"devicetype": "Ref","stage": "PROD","platform": "RT 1.0","di": "d363022f-a05f-be2b-0d11-79f783d308df","uid": "e8rnuugabc","serialnumber": "343333443333335","macaddress": "999999999999","mnmo": "ARTIK051|00108441|000008500012115A0100000000000000"},"ipaddress":"", "versioninfo": {"wifi": "991120190702","firmware": "99121706, FFFFFF"},"eventlist": [{"name": "E01","dt": "2019-08-25 22:36:02"}]}
filter {
```
split {
field => "eventlist"
}
fingerprint {
target => "[@metadata][fingerprint]"
source => ["deviceinfo[di]","@timestamp","eventlist[name]"]
#source => ["eventlist[name]"]
method => "SHA1"
key => "Log analytics"
base64encode => true
concatenate_sources => true
}
json {
source => "message"
target => ""
}
date {
match => ["eventlist[dt]", "YYYY-MM-dd HH:mm:ss"]
}
mutate {
remove_field => ["eventlist[logencoding]","eventlist[dt]","@version"]
}
```
}
When above duplicate entries of sample data is read from same file, duplicacy is removed but if read from different file, duplicacy remains and we need to read data from different files and eliminate duplicacy.
Thanks
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 24, 2019, 4:06am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663/4 "2019-09-24T04:06:58Z")
Are all the fields that make up the hash identical? Can you show an example of two duplicate events that have been indexed?
How come you are running the JSON filter after fingerprint? Does the fields even exist when the fingerprint filter is run?
---
### Author:  [@sharry007](https://discuss.elastic.co/u/sharry007)
#### Post date: [September 24, 2019, 4:14am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663/5 "2019-09-24T04:14:30Z")
> [@Christian\_Dahlqvist](#):
>
> Are all the fields that make up the hash identical? Can you show an example of two duplicate events that have been indexed?
>
> How come you are running the JSON filter after fingerprint? Does the fields even exist when the fingerprint filter is run?
Yes, those are identical. Below is an example of duplicate data
Time @timestamp deviceinfo.di eventlist.name \_id
Aug 26, 2019 @ 04:06:02.000 Aug 26, 2019 @ 04:06:02.000 d363022f-a05f-be2b-0d11-79f783d308df E01 8bl+kCq78C2adJis6lIAaLFimPo=
Aug 26, 2019 @ 04:06:02.000 Aug 26, 2019 @ 04:06:02.000 d363022f-a05f-be2b-0d11-79f783d308df E01 X3rut13vaNd3fd2NzjNLfX+V+pg=
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 24, 2019, 4:20am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663/6 "2019-09-24T04:20:09Z")
Please show full indexed events from Elasticsearch in JSON. Make sure the document id is shown.
What does the JSON filter parse out from the message field?
---
### Author:  [@sharry007](https://discuss.elastic.co/u/sharry007)
#### Post date: [September 24, 2019, 4:34am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663/7 "2019-09-24T04:34:16Z")
{
"\_index": "logs-2019.08.25",
"\_type": "\_doc",
"\_id": "8bl+kCq78C2adJis6lIAaLFimPo=",
"\_version": 1,
"\_score": null,
"\_source": {
"servicename": "Ref",
"versioninfo": {
"wifi": "991120190702",
"firmware": "99121706, FFFFFF"
},
"logtype": "logs",
"@timestamp": "2019-08-25T22:36:02.000Z",
"ipaddress": "",
"deviceinfo": {
"macaddress": "999999999999",
"di": "d363022f-a05f-be2b-0d11-79f783d308df",
"stage": "PROD",
"productid": "ARTIK051",
"platform": "RT 1.0",
"uid": "e8rnuugabc",
"serialnumber": "343333443333335",
"devicetype": "Ref",
"mnmo": "ARTIK051\_REF\_17K|00108441|000008500012115A0100000000000000",
"policyversion": 1541130000
},
"eventlist": {
"name": "ES01"
}
},
"fields": {
"@timestamp": [
"2019-08-25T22:36:02.000Z"
]
},
"sort": [
1566772562000
]
}
{
"\_index": "logs-2019.08.25",
"\_type": "\_doc",
"\_id": "X3rut13vaNd3fd2NzjNLfX+V+pg=",
"\_version": 1,
"\_score": null,
"\_source": {
"servicename": "Ref",
"versioninfo": {
"wifi": "991120190702",
"firmware": "99121706, FFFFFF"
},
"logtype": "logs",
"@timestamp": "2019-08-25T22:36:02.000Z",
"ipaddress": "",
"deviceinfo": {
"macaddress": "999999999999",
"di": "d363022f-a05f-be2b-0d11-79f783d308df",
"stage": "PROD",
"productid": "ARTIK051",
"platform": "RT 1.0",
"uid": "e8rnuugabc",
"serialnumber": "343333443333335",
"devicetype": "Ref",
"mnmo": "ARTIK051\_REF\_17K|00108441|000008500012115A0100000000000000",
"policyversion": 1541130000
},
"eventlist": {
"name": "ES01"
}
},
"fields": {
"@timestamp": [
"2019-08-25T22:36:02.000Z"
]
},
"sort": [
1566772562000
]
}
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 24, 2019, 5:45am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663/8 "2019-09-24T05:45:24Z")
What does the JSON filter do? Are the fields available when fingerprint is run?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 22, 2019, 5:45am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663/9 "2019-10-22T05:45:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.