# Logstash fingerprint 7.1
**URL:** https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598
**Category:** Logstash
**Created:** [June 6, 2019, 1:53pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598 "2019-06-06T13:53:10Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@pgervais](https://discuss.elastic.co/u/pgervais)
#### Post date: [June 6, 2019, 1:53pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598/1 "2019-06-06T13:53:10Z")
I want to ensure that the document id are based on the input source. For this i'm using the fingerprint filter in logstash. In previous version i've used it as follows
```
fingerprint {
source => "message"
target => "[@metadata][fingerprint]"
method => "SHA1"
key => "Websphere Metrics by Tran Class"
base64encode => true
}
```
With the corresponding output section below:
output {
elasticsearch {
action =\> "index"
hosts =\> "localhost:9200"
index =\> "websphere"
document\_id =\> "%{[@metadata][fingerprint]}"
```
}
```
# stdout {codec =\> rubydebug}
# stdout {}
}
I have 4603 document i'm trying to ingest into the index websphere.
What I get now is one entry i.e. the last entry is the only one that survives.
If I remove the fingerprint , I get all document id.
1)Is this use of fingerprint not backwards compatible?
2) I have also tried to set the "concatenate\_sources" =\> true with no change.
What is the proper way to do this duplicates removal in 7.1? Essentially i'm looking for the old behaviour in 6.0 for 7.1.
---
### Author:  [@BennyInc](https://discuss.elastic.co/u/BennyInc)
#### Post date: [June 6, 2019, 2:05pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598/2 "2019-06-06T14:05:50Z")
What do your message fields look like? Are they identical by chance?
Maybe you can include the timestamp of the message in the fingerprint?
You could give it an array of `["message","@timestamp"]` and use concatenate\_sources then?
Alternatively try [concatenate\_all\_fields](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html#plugins-filters-fingerprint-concatenate_all_fields)?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 6, 2019, 3:33pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598/3 "2019-06-06T15:33:26Z")
> [@pgervais](#):
>
> What I get now is one entry i.e. the last entry is the only one that survives.
What is the document id on that entry?
---
### Author:  [@pgervais](https://discuss.elastic.co/u/pgervais)
#### Post date: [June 6, 2019, 3:39pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598/4 "2019-06-06T15:39:17Z")
The message field is input from a jdbc connection.
I have tried the concatenate\_all\_fields as well as concatenate\_sources.
So I do one logstash run. Look at the discover panel and see that the last data point is made up of 78 entries. Then I run the same file again.
If fingerprint works , the counts for each entry should not change i.e. they will be simply written on toop of each other. Counts after second run : 156 i.e. 78\*2.
The complete logstash file is shown below: # This config file is used to parse the sql data extracted from the db entry:PostalMod
input {
jdbc {
jdbc\_driver\_library =\> "/home/pxg110/sqljdbc\_4.2/sqljdbc42.jar"
jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc\_user =\> "CBSA\_WLM\_SVCg"
jdbc\_password =\> "CBSA\_WLM\_SVCg"
lowercase\_column\_names =\> "false"
jdbc\_connection\_string =\> "jdbc:sqlserver://SD01CUVDB0521.OMEGA.DCE-EIR.NET:1433;"
statement =\> "SELECT ObsDate,ObsHour,TotalCPULoadMIPS,GPPLoadMIPS,zIIPLoadMIPS,GPPPathlenMilsInstr,zIIPPathlenMilsInstr,AvgNetworkTrafficKBsec FROM smg.dbo.smgdata WHERE (ApplnName='PostalMod') AND (ObsDate \>= CONVERT(DATETIME, '2019-03-23', 102)) AND (ObsDate \<= CONVERT(DATETIME, '2019-05-23', 102)) ORDER BY ObsDate;"
}
}
filter {
fingerprint {
source =\> "message"
target =\> "[@metadata][fingerprint]"
concatenate\_all\_fields =\> "true"
method =\> "SHA1"
key =\> "wed\_jun\_2019\_11\_58\_postalmod"
base64encode =\> true
}
# defines all the fields to be found in the csv file.
```
csv {
separator => ","
columns => [
"ObsDate",
"ObsHour",
"TotalCPULoadMIPS",
"GPPLoadMIPS",
"zIIPLoadMIPS",
"GPPPathlenMilsInstr",
"zIIPPathlenMilsInstr",
"AvgNetworkTrafficKBsec"
]
convert => {
"ObsDate" => "date"
"ObsHour" => "integer"
"TotalCPULoadMIPS" => "float"
"GPPLoadMIPS" => "float"
"zIIPLoadMIPS" => "float"
"GPPPathlenMilsInstr" => "float"
"zIIPPathlenMilsInstr" => "float"
"AvgNetworkTrafficKBsec" => "float"
}
}
```
A typical output is shown below:
{
"TotalCPULoadMIPS" =\> 29.07,
"AvgNetworkTrafficKBsec" =\> 1.98,
"zIIPPathlenMilsInstr" =\> 43.41,
"GPPLoadMIPS" =\> 0.175,
"zIIPLoadMIPS" =\> 28.89,
"@version" =\> "1",
"GPPPathlenMilsInstr" =\> 0.26,
"@timestamp" =\> 2019-03-24T18:00:00.000Z
}
The timestamp matches the date on the sql results. No datetimeparse error.
```
mutate {
convert => { "ObsDate" => "string" }
}
dissect {
mapping => {
"ObsDate" => "%{year}-%{month}-%{day}T%{hour}:%{minute}:%{seconds}.%{ms}Z"
}
}
```
# ISO time stamp 2011-04-19T03:44:01.103Z
```
mutate {
add_field => { "timestamp" => "%{year}-%{month}-%{day}:%{ObsHour}" }
}
date {
match => ["timestamp", "yyyy-MM-dd:HH"]
target => "@timestamp"
}
mutate {
remove_field => ["ObsDate", "ObsHour", "year","month", "day","hour","minute","seconds","ms","timestamp"]
}
```
}
output {
elasticsearch {
action =\> "index"
hosts =\> "localhost:9200"
document\_id =\> "%{[@metadata][fingerprint]}"
index =\> "wed\_jun\_2019\_11\_58\_postalmod"
}
stdout {codec =\> rubydebug}
}
---
### Author:  [@BennyInc](https://discuss.elastic.co/u/BennyInc)
#### Post date: [June 6, 2019, 4:04pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598/5 "2019-06-06T16:04:33Z")
So, do you even have a `message` field?
Can you add a file output and check the output JSON for what exact data is extracted on your runs?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 6, 2019, 4:48pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598/6 "2019-06-06T16:48:31Z")
If you do not have a message field then I would expect all of the documents to have the document id "ovLqdkaUAOyjWzfsW9WXJqjwuew="
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 4, 2019, 4:56pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598/7 "2019-07-04T16:56:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.