# FScrawler - Duplicate Files

**URL:** <https://discuss.elastic.co/t/fscrawler-duplicate-files/194322>\
**Category:** Elasticsearch\
**Created:** [August 7, 2019, 8:32pm UTC](https://discuss.elastic.co/t/fscrawler-duplicate-files/194322 "2019-08-07T20:32:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bowriverstudio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bowriverstudio/32/51887_2.png) [@Bowriverstudio](https://discuss.elastic.co/u/Bowriverstudio)\
**Post date:** [August 7, 2019, 8:32pm UTC](https://discuss.elastic.co/t/fscrawler-duplicate-files/194322/1 "2019-08-07T20:32:01Z")

</div>

Hello All,

I am a newbie to elastic search.

I am interested in using FScrawler to index all my files, then Elasticsearch find duplicates.

I'm thinking of using the md5 function to hash all the files: [https://fscrawler.readthedocs.io/en/fscrawler-2.5/admin/fs/local-fs.html#file-checksum](https://fscrawler.readthedocs.io/en/fscrawler-2.5/admin/fs/local-fs.html#file-checksum)

Then use something like [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html) to search for duplicates.

Or would something like this be a better approach.

> **[How to Find and Remove Duplicate Documents in Elasticsearch](https://www.elastic.co/blog/how-to-find-and-remove-duplicate-documents-in-elasticsearch)**
>
> Learn how to detect and remove duplicate documents from Elasticsearch using Logstash or a custom Python script.

Or is there a better way?

Cheers,  
Maurice

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 10, 2019, 4:03pm UTC](https://discuss.elastic.co/t/fscrawler-duplicate-files/194322/2 "2019-08-10T16:03:58Z")

</div>

Hey Maurice

It could be easier not to index duplicated files in the first place.

Would that be an option?

Otherwise, I think the last link you shared is the good way.

---

<div class="post-metadata">

**Author:** ![Bowriverstudio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bowriverstudio/32/51887_2.png) [@Bowriverstudio](https://discuss.elastic.co/u/Bowriverstudio)\
**Post date:** [August 12, 2019, 4:03pm UTC](https://discuss.elastic.co/t/fscrawler-duplicate-files/194322/3 "2019-08-12T16:03:14Z")

</div>

Unfortunately I do not control the source of the files being indexed. Thank you for the response.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 9, 2019, 4:03pm UTC](https://discuss.elastic.co/t/fscrawler-duplicate-files/194322/4 "2019-09-09T16:03:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.