# How to ignore the same log contents in the different log files?
**URL:** https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757
**Category:** Logstash
**Created:** [May 14, 2018, 2:18pm UTC](https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757 "2018-05-14T14:18:49Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@alicia1](https://discuss.elastic.co/u/alicia1)
#### Post date: [May 14, 2018, 2:18pm UTC](https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757/1 "2018-05-14T14:18:49Z")
Hi,
We have logs files in different repertories, same logs contents can find several times in different files.
How to ignore the same log contents in the different log files in order to send only once a same log content to ES?
Thanks.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 15, 2018, 8:45pm UTC](https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757/2 "2018-05-15T20:45:16Z")
Logstash has no standard plugins to de-duplicate logs, but if you compute the Elasticsearch document id based on the message contents you can de-duplicate on the ES side (i.e. you'll send all documents to ES but the duplicates will overwrite each other, leaving a single copy of each).
---
### Author:  [@alicia1](https://discuss.elastic.co/u/alicia1)
#### Post date: [May 16, 2018, 9:48am UTC](https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757/3 "2018-05-16T09:48:41Z")
Hi @magnusbaeck,
Thanks for your reply.
Sorry, i don't understand, how i can do this on the ES side?
You talk about use DSL query in Kibana?
Thanks.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 16, 2018, 12:58pm UTC](https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757/4 "2018-05-16T12:58:14Z")
No, I meant that you use the fingerprint filter to compute a checksum of the event, store that checksum in a field that you reference in the `document_id` option of your elasticsearch output. If two identical events arrive they'll get the same document id and therefore won't be stored twice.
---
### Author:  [@alicia1](https://discuss.elastic.co/u/alicia1)
#### Post date: [May 16, 2018, 2:40pm UTC](https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757/5 "2018-05-16T14:40:23Z")
Thanks @magnusbaeck.
Do you have an example about that config?
Thanks.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 13, 2018, 2:40pm UTC](https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757/6 "2018-06-13T14:40:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.