# File input duplicates **URL:** \ **Category:** Logstash\ **Created:** [May 15, 2019, 11:51am UTC](https://discuss.elastic.co/t/file-input-duplicates/181169 "2019-05-15T11:51:54Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Sali7007](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@Sali7007](https://discuss.elastic.co/u/Sali7007)\ **Post date:** [May 15, 2019, 11:51am UTC](https://discuss.elastic.co/t/file-input-duplicates/181169/1 "2019-05-15T11:51:54Z")
Hi I have a file which is rewritten daily with new data. Everything was working fine till I missed to update the file with new data. So, Logstash processed the old data itself again generating duplicate data. Is there a way that i can correct that(remove duplicate data)? Is there a way to avoid this issue in future? ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [May 15, 2019, 12:13pm UTC](https://discuss.elastic.co/t/file-input-duplicates/181169/2 "2019-05-15T12:13:21Z")
> [@Sali7007](#): > > old data itself again generating Yes, there is a way. Have a look on the fingerprint filter and MD5 hash calculation. You calculate MD5 out of your message and later use it in the output as the document id when loading the data to elasticsearch. If you want to remove duplicate data: - read the data from the elasticsearch in logstash (input elasticsearch) - in filter section calculate md5 out of the fields you want to have unique id - output to the elasticsearch with the document id calculated in a fingerprint section. Cheers. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 12, 2019, 12:13pm UTC](https://discuss.elastic.co/t/file-input-duplicates/181169/3 "2019-06-12T12:13:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.