# How we can remove deduplication event in logstash
**URL:** https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060
**Category:** Elasticsearch
**Created:** [August 3, 2023, 2:52pm UTC](https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060 "2023-08-03T14:52:39Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Sukhdeob\_95](https://discuss.elastic.co/u/Sukhdeob_95)
#### Post date: [August 3, 2023, 2:52pm UTC](https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060/1 "2023-08-03T14:52:39Z")
I want to remove the duplicate event based on particular field of my input
i wrote logic like following but i got an error
aggregate {
task\_id =\> "%{[meta][ingestionHash]}"
code =\> "
map['@metadata']['keep'] ||= event.get('[@metadata][first\_event]') ? false : true
event.set('[@metadata][first\_event]', true)
"
end\_of\_task =\> true
}
if ![[@metadata][keep]] {
drop { }
}
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [August 4, 2023, 4:42am UTC](https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060/2 "2023-08-04T04:42:59Z")
Welcome to the community
Usually, deduplication is done by the fingerprint plugin. Check [the blog](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)
---
### Author:  [@Sukhdeob\_95](https://discuss.elastic.co/u/Sukhdeob_95)
#### Post date: [August 4, 2023, 5:24am UTC](https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060/3 "2023-08-04T05:24:23Z")
can we filter out based on some field value.
---
### Author:  [@Sukhdeob\_95](https://discuss.elastic.co/u/Sukhdeob_95)
#### Post date: [August 4, 2023, 5:27am UTC](https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060/4 "2023-08-04T05:27:32Z")
In input i have a field name "ingestionHash" and based on this field i want to remove duplicate event. if two events having the same ingestionHash value then i want ignore the second event..
aggregate {
task\_id =\> "%{[meta][ingestionHash]}"
code =\> "
map['@metadata']['keep'] ||= event.get('[@metadata][first\_event]') ? false : true
event.set('[@metadata][first\_event]', true)
"
end\_of\_task =\> true
}
if ![[@metadata][keep]] {
drop { }
}
this is my code for remove deduplicate event . is it correct ?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 4, 2023, 1:17pm UTC](https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060/5 "2023-08-04T13:17:27Z")
> [@Sukhdeob\_95](#):
>
> In input i have a field name "ingestionHash" and based on this field i want to remove duplicate
Why not use this field as the `document_id` in your elasticsearch output?
The deduplication needs to be done in Elasticsearch not Logstash.
I do not use the aggregation filter, so I can not tell if your code is right or not, but I do not think that even if it is right it will work as you want.
The aggregate filter has a timeout, a time range in which it will agregate the events, if you receive two events with the same `ingestionHash` value, but they come outside this time range, the aggregate filter will do nothing.
For example, if you receive an event with the `ingestionHash` value of `abcd-1234` now and receive the same `ingestionHash` 15 minutes later, they will note be aggregate because the timeout for the `aggregate` filter will already be expired.
---
### Author:  [@Sukhdeob\_95](https://discuss.elastic.co/u/Sukhdeob_95)
#### Post date: [August 4, 2023, 4:53pm UTC](https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060/6 "2023-08-04T16:53:48Z")
instead of using document\_id, is there any another way to remove deduplicate event based on some field value. because to create a document\_id i have use fingerprint filter plugin.
following is my config file
input {
}
filter {
mutate{
split =\> ["topicParts", "."]
add\_field =\> { "dataType" =\> "%{[topicParts][2]}" }
add\_field =\> { "ingestKeyHash" =\> "" }
}
fingerprint {
source =\> ["[meta][recordKeys]"]
method =\> "MURMUR3"
target =\> "ingestKeyHash"
}
}
output {
opensearch {
hosts =\> ["${ELASTIC\_SEARCH\_URL}:443"]
auth\_type =\> {
type =\> 'aws\_iam'
aws\_access\_key\_id =\> '${LOGSTASH\_USER\_ACCESS\_KEY}'
aws\_secret\_access\_key =\> '${LOGSTASH\_USER\_SECRET\_KEY}'
region =\> 'us-west-2'
}
ecs\_compatibility =\> disabled
index =\> "logstash-%{[@metadata][kafka][topic]}-%{+YYYY.MM}"
ssl\_certificate\_verification =\> true
document\_id =\> "%{[@metadata][kafka][topic]}-%{[ingestKeyHash]}"
}
}
input data 👎
{
"topicTimestamp": "2023-07-26T07:33:27.032Z",
"ingestKeyHash": 3171307072,
"collectionSource": "service.mds.ro",
"partition": "0",
"dataType": "avro",
"@version": "1",
"@timestamp": "2023-07-26T07:33:27.048Z",
"offset": "333",
"meta": {
"providerDealerId": "dealer1",
"dataType": null,
"ingestionHash": "421047889",
"providerName": "Drive DMS",
"subscriptionPartnerName": null,
"recordKeys": {
"roNum": "134825",
"providerDealerId": "3P12we",
"providerId": "a41a5ea7",
"roOpenDate": "2023-07-25"
}
}
}
Note -: ingestKeyHash and ingestionHash are two different fields.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 4, 2023, 4:53pm UTC](https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060/7 "2023-08-04T16:53:48Z")
OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.
(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 1, 2023, 4:53pm UTC](https://discuss.elastic.co/t/how-we-can-remove-deduplication-event-in-logstash/340060/8 "2023-09-01T16:53:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.