# When indexing a repository of files (using fsCrawler or genearlly) what happens under the hood? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 3, 2020, 3:57pm UTC](https://discuss.elastic.co/t/when-indexing-a-repository-of-files-using-fscrawler-or-genearlly-what-happens-under-the-hood/230885 "2020-05-03T15:57:15Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Lisahtwy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisahtwy/32/53910_2.png) [@Lisahtwy](https://discuss.elastic.co/u/Lisahtwy)\ **Post date:** [May 3, 2020, 3:57pm UTC](https://discuss.elastic.co/t/when-indexing-a-repository-of-files-using-fscrawler-or-genearlly-what-happens-under-the-hood/230885/1 "2020-05-03T15:57:15Z")
This might seem silly, but What happens when I index files in a repository? just want to elaborate my question here.. I have around 20GB of files, when I indexed them using fsCrawler it created an index of size 56mb. (stats from `GET _cat/indices?`) I ran a search query to find a name, lets say "Sam". In the results I can see full file contents in the "hits\>source\>content". My question is how can you convert a 20GB repository to 56mb index with full file contents? what happens when I index a file or repository of files? Could you please explain or point me to any documentation on this topic. I searched on the official docs but couldnt find any explanation on this. -Lisa ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 4, 2020, 8:40am UTC](https://discuss.elastic.co/t/when-indexing-a-repository-of-files-using-fscrawler-or-genearlly-what-happens-under-the-hood/230885/2 "2020-05-04T08:40:41Z")
FSCrawler extracts text and some metadata from a file. Imagine you are crawling a video file of 500mb, no text will be extracted but just few metadata. It means that you are going to index may be 1kb on 500mb. Same goes with PDF or Word documents which has pictures. The pictures won't add any text but contributes to the original size. That could explain a lot what you are seeing. ---
**Author:** ![Lisahtwy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisahtwy/32/53910_2.png) [@Lisahtwy](https://discuss.elastic.co/u/Lisahtwy)\ **Post date:** [May 9, 2020, 12:48am UTC](https://discuss.elastic.co/t/when-indexing-a-repository-of-files-using-fscrawler-or-genearlly-what-happens-under-the-hood/230885/3 "2020-05-09T00:48:48Z")
Hi, What kind of metadata would fsCrawler extract ? Can I check anywhere? I mean using any API call or something? is there any document for more details? -Lisa ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 9, 2020, 3:35am UTC](https://discuss.elastic.co/t/when-indexing-a-repository-of-files-using-fscrawler-or-genearlly-what-happens-under-the-hood/230885/4 "2020-05-09T03:35:32Z")
Things like title, tags.... See [https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#generated-fields](https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#generated-fields) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 6, 2020, 3:35am UTC](https://discuss.elastic.co/t/when-indexing-a-repository-of-files-using-fscrawler-or-genearlly-what-happens-under-the-hood/230885/5 "2020-06-06T03:35:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.