# Efficient Metadata Indexing for Large Filesystem in Elasticsearch

**URL:** <https://discuss.elastic.co/t/efficient-metadata-indexing-for-large-filesystem-in-elasticsearch/357981>\
**Category:** Elasticsearch\
**Created:** [April 23, 2024, 6:41am UTC](https://discuss.elastic.co/t/efficient-metadata-indexing-for-large-filesystem-in-elasticsearch/357981 "2024-04-23T06:41:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![geekahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geekahmed/32/133752_2.png) [@geekahmed](https://discuss.elastic.co/u/geekahmed)\
**Post date:** [April 23, 2024, 6:41am UTC](https://discuss.elastic.co/t/efficient-metadata-indexing-for-large-filesystem-in-elasticsearch/357981/1 "2024-04-23T06:41:08Z")

</div>

Hello everyone,

I'm working with a mounted network filesystem on Linux that contains a massive collection of files (text, images, videos, etc.). My goal is to index specific metadata from these files into Elasticsearch.

Specifically, I need the following fields:

- Filename
- Creation Date
- File Path
- Extension

I'm not interested in content indexing and only require the listed metadata. Additionally, certain folders within the filesystem will be updated over time.

I've explored FSCrawler for this task but found it doesn't scale well with large numbers of files and tends to get stuck. I'm looking for a more robust solution that can:

- Monitor specific folders
- Index metadata of new files as they're added
- Handle a massive volume of files efficiently

Any guidance or recommendations would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 23, 2024, 12:19pm UTC](https://discuss.elastic.co/t/efficient-metadata-indexing-for-large-filesystem-in-elasticsearch/357981/2 "2024-04-23T12:19:25Z")

</div>

Welcome!

FWIW with FSCrawler, it does not scale well indeed the way it's implemented today.  
The workaround is to create more than one FSCrawler instance, let say one per dir on the root dir.

---

<div class="post-metadata">

**Author:** ![geekahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geekahmed/32/133752_2.png) [@geekahmed](https://discuss.elastic.co/u/geekahmed)\
**Post date:** [April 23, 2024, 2:05pm UTC](https://discuss.elastic.co/t/efficient-metadata-indexing-for-large-filesystem-in-elasticsearch/357981/3 "2024-04-23T14:05:57Z")

</div>

I've been experimenting with FSCrawler and found it can index ~18 million files in 17 hours. To optimize this, I'm considering a hybrid approach:

1. **Pre-indexing static files:** Use FSCrawler with includes/excludes regex to index known, unchanging content.
2. **Scripted FSCrawler for dynamic folders:** Since I know when new folders with data will be added (and that these periods are short, e.g., 5 weeks), I'd like to use a script to generate FSCrawler settings and trigger it automatically before data is added.

**Question:** Do you think this hybrid approach is a reasonable solution? Are there potential pitfalls or better alternatives I should consider?