# \[Threat Intelligence\]: Avoid redundancy of information in the same index
**URL:** https://discuss.elastic.co/t/threat-intelligence-avoid-redundancy-of-information-in-the-same-index/260303
**Category:** Logstash
**Created:** [January 6, 2021, 9:04am UTC](https://discuss.elastic.co/t/threat-intelligence-avoid-redundancy-of-information-in-the-same-index/260303 "2021-01-06T09:04:54Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [January 6, 2021, 9:04am UTC](https://discuss.elastic.co/t/threat-intelligence-avoid-redundancy-of-information-in-the-same-index/260303/1 "2021-01-06T09:04:54Z")
Hello,
I am using this pipeline to enrich my SIEM with URLhaus information:
```auto
input {
exec {
command => 'curl https://urlhaus.abuse.ch/downloads/csv/ --output text.zip && unzip -c text.zip'
interval => 86400
type => 'iphaus'
codec => line
}
}
filter {
if [type] == "iphaus" {
csv {
columns => ["id","dateadded","url","url_status","threat","tags","urlhaus_link","reporter"]
separator => ","
}
mutate {
remove_field => ["message"]
}
}
}
output {
elasticsearch {
hosts => ["https://X.X.X.X:9200"]
index => "malware-%{+YYYY.MM.dd}"
cacert => 'ca.crt'
user => "elastic"
password => "password"
}
}
```
The only problem is that by downloading it every 24h, I am getting redundancy in my index, ( the same URL is present in the database of yesterday and of today).
I would like to know how can I match the url to my index url field, to see if it's already present or not, and then index it if it's not already indexed.
Thanks for your help.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 6, 2021, 4:31pm UTC](https://discuss.elastic.co/t/threat-intelligence-avoid-redundancy-of-information-in-the-same-index/260303/2 "2021-01-06T16:31:20Z")
If you set the document\_id on the elasticsearch output based on some field, or a fingerprint generated from multiple fields of the document, then the elasticsearch output will overwrite the document instead of inserting a duplicate.
---
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [January 6, 2021, 4:37pm UTC](https://discuss.elastic.co/t/threat-intelligence-avoid-redundancy-of-information-in-the-same-index/260303/3 "2021-01-06T16:37:21Z")
Thanks for your answer @Badger,
Could you tell me please how can I set that in my Logstash configuration !
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 6, 2021, 4:50pm UTC](https://discuss.elastic.co/t/threat-intelligence-avoid-redundancy-of-information-in-the-same-index/260303/4 "2021-01-06T16:50:14Z")
To generate a fingerprint use a filter
```
fingerprint {
concatenate_sources => true
method => "SHA256"
source => ["url"] # And possibly other fields
target => "[@metadata][fingerprint]"
}
```
then reference it in the elasticsearch output using a sprintf reference
```
document_id => "%{[@metadata][fingerprint]}"
```
---
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [January 7, 2021, 8:11am UTC](https://discuss.elastic.co/t/threat-intelligence-avoid-redundancy-of-information-in-the-same-index/260303/5 "2021-01-07T08:11:59Z")
Thank you very much @Badger,
I just tried it and it's working like a charm 🤩
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 4, 2021, 8:12am UTC](https://discuss.elastic.co/t/threat-intelligence-avoid-redundancy-of-information-in-the-same-index/260303/6 "2021-02-04T08:12:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.