# Fingerprint option in logstash filter not working properly?
**URL:** https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115
**Category:** Logstash
**Created:** [January 22, 2022, 12:23pm UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115 "2022-01-22T12:23:37Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Anjali\_Kushwaha](https://discuss.elastic.co/u/Anjali_Kushwaha)
#### Post date: [January 22, 2022, 12:23pm UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/1 "2022-01-22T12:23:37Z")
I have used below code but its only inserting one record to ES .
```auto
filter {
fingerprint {
target => "document_id"
method => "SHA256"
key => "9ced3827c6a1c9dafac6da9abac41386ba1038ac95b3a865a0951bc2e948c58c"
}
}
output {
elasticsearch {
host => localhost
document_id => "%{document_id}"
}
}
```
Please point out the mistake below I have approx 2 lac records in my sql which i am importing to ES , I want unique ids for each record using this fingerprint.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 22, 2022, 1:27pm UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/2 "2022-01-22T13:27:04Z")
I don't know the answer but you can also not set the id by yourself and let Elasticsearch decide about it.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [January 22, 2022, 2:14pm UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/3 "2022-01-22T14:14:57Z")
Hello,
Can you explain what is not working? What is your output and what do you expect?
Also, since you are not setting the `source` option in the fingerprint filter, it will use the `message` field, do you have a message field in the event? You didn't share the entire pipeline, so there is no way to know.
What is 2 lac? It is 2 k? 200 k? 2 millions?
---
### Author:  [@Anjali\_Kushwaha](https://discuss.elastic.co/u/Anjali_Kushwaha)
#### Post date: [January 24, 2022, 4:31am UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/4 "2022-01-24T04:31:21Z")
Can you please tell me how can i insert 2,00,000 records uniquely to ES using fingerprint?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 24, 2022, 4:47am UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/5 "2022-01-24T04:47:27Z")
Unless you have a message field to base the fingerprint on you need to specify the source field(s) to use in the plugin. It would help if you showed your full pipeline as well as a sample event.
---
### Author:  [@Anjali\_Kushwaha](https://discuss.elastic.co/u/Anjali_Kushwaha)
#### Post date: [January 24, 2022, 4:59am UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/6 "2022-01-24T04:59:32Z")
Sample data -
MON\_NO CODE SUM DATE YEAR\_NO TIME\_DOC
|APR|SUN0455|58639.5|2021-04-01 00:00:00|2021|2021-04-01 00:00:00|
|APR|SUN0455|58639.5|2021-04-01 00:00:00|2021|2021-04-01 00:00:00|
|MAY|SUN0457|639.5|2021-05-01 00:00:00|2021|2021-05-01 00:00:00|
|APR|SUN0455|58639.5|2021-04-01 00:00:00|2021|2021-04-01 00:00:00|
|APR|SUN0455|58639.5|2021-04-01 00:00:00|2021|2021-04-01 00:00:00|
input{
jdbc{
jdbc {
# Oracle
jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver"
statement\_filepath =\> "folio\_aum.sql"
```
}
```
}
filter {
fingerprint {
target =\> "document\_id"
method =\> "SHA256"
key =\> "9ced3827c6a1c9dafac6da9abac41386ba1038ac95b3a865a0951bc2e948c58c"
}
}
output {
Elasticsearch {
host =\> localhost
document\_id =\> "%{document\_id}"
}
}
Please help in sending this data uniquely into ES
---
### Author:  [@Anjali\_Kushwaha](https://discuss.elastic.co/u/Anjali_Kushwaha)
#### Post date: [January 24, 2022, 5:01am UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/7 "2022-01-24T05:01:39Z")
@Christian_Dahlqvist can you help as per below sample data i posted?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 24, 2022, 5:02am UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/8 "2022-01-24T05:02:41Z")
WhT does a sample document generated by the jdbc input plugin look like? Which of these fields can be used to uniquely identify an event/document?
---
### Author:  [@Anjali\_Kushwaha](https://discuss.elastic.co/u/Anjali_Kushwaha)
#### Post date: [January 24, 2022, 5:08am UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/9 "2022-01-24T05:08:11Z")
@Christian_Dahlqvist JDBC output is same as I shared the sample data , I am basically getting the data from SQL and importing to ES, We can take any number of fields, like this I have 15 columns , How can I use fingerprint to generate a unique id for each record, as example we have UUID in python which generates unique id for each record, similarly how to use fingerprint to generate unique id?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 24, 2022, 5:39am UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/10 "2022-01-24T05:39:37Z")
Please show the sample document in JSON form as it looks when inserted into Elasticsearch. This will show the fields available, which depends on the SQL statement.
The fingerprint filter allows you to generate a hash based on some fields in the data that forms a primary key, which can be used as an ID. If you are running the JDBC query a number of times this will prevent duplicates from being created.
You could instead set the fingerprint plugin to generate a UUID, but that would be unique every time it runs for a record. If you reprocess the same data through the JDBC plugin that data would get inserted multiple times. Rather than using a UUID it would be more efficient to let Elasticsearch assign the ID.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 21, 2022, 5:40am UTC](https://discuss.elastic.co/t/fingerprint-option-in-logstash-filter-not-working-properly/295115/11 "2022-02-21T05:40:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.