# If FSCrawler stops mid-scan, will it restart from scratch?

**URL:** <https://discuss.elastic.co/t/if-fscrawler-stops-mid-scan-will-it-restart-from-scratch/373787>\
**Category:** Elasticsearch\
**Created:** [January 28, 2025, 6:25pm UTC](https://discuss.elastic.co/t/if-fscrawler-stops-mid-scan-will-it-restart-from-scratch/373787 "2025-01-28T18:25:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![syoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syoo/32/140632_2.png) [@syoo](https://discuss.elastic.co/u/syoo)\
**Post date:** [January 28, 2025, 6:25pm UTC](https://discuss.elastic.co/t/if-fscrawler-stops-mid-scan-will-it-restart-from-scratch/373787/1 "2025-01-28T18:25:48Z")

</div>

As a newcomer to Elasticsearch, I want to index a large filesystem (100TB) with FSCrawler.

I’ve heard that if the initial scan is interrupted, FSCrawler might re-scan everything from the beginning.

What’s the best strategy to handle this for such a massive dataset?

Are there checkpointing/resume features, configuration tweaks, or alternative workflows to avoid redundant work?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 28, 2025, 9:04pm UTC](https://discuss.elastic.co/t/if-fscrawler-stops-mid-scan-will-it-restart-from-scratch/373787/2 "2025-01-28T21:04:37Z")

</div>

Welcome!

This is correct and until a change is made the only workaround I can see is to start multiple fscrawler instances, one per dir in the root directory.

---

<div class="post-metadata">

**Author:** ![syoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syoo/32/140632_2.png) [@syoo](https://discuss.elastic.co/u/syoo)\
**Post date:** [January 28, 2025, 9:27pm UTC](https://discuss.elastic.co/t/if-fscrawler-stops-mid-scan-will-it-restart-from-scratch/373787/3 "2025-01-28T21:27:52Z")

</div>

> [@dadoonet](#):
>
> Welcome!
> 
> This is correct and until a change is made the only workaround I can see is to start multiple fscrawler instances, one per dir in the root directory.

Thank you for your support. Then once I scan all the files using multiple FSCrawler instances, will running FSCrawler on the root directory continue tracking changes without performing a full re-scan?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 29, 2025, 1:15am UTC](https://discuss.elastic.co/t/if-fscrawler-stops-mid-scan-will-it-restart-from-scratch/373787/4 "2025-01-29T01:15:27Z")

</div>

I think you will need to keep it running as it was running for the first run.  
I said "I think" because I'm not sure about it. I don't remember how ids are computed.

May be you could run it from the root and use different includes settings for each instance. And then run again from root without the includes setting... That way, all ids should be consistent.