# How to solve index millions of files content with scalable approach? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 24, 2022, 7:25am UTC](https://discuss.elastic.co/t/how-to-solve-index-millions-of-files-content-with-scalable-approach/303084 "2022-04-24T07:25:54Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![sushilprj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sushilprj/32/11593_2.png) [@sushilprj](https://discuss.elastic.co/u/sushilprj)\ **Post date:** [April 24, 2022, 7:25am UTC](https://discuss.elastic.co/t/how-to-solve-index-millions-of-files-content-with-scalable-approach/303084/1 "2022-04-24T07:25:54Z")
I am working with an application that requires indexing thousands of files' content to make the content searchable. I use tika to extract the file content and then index it to Elasticsearch. I also tried to ingest pipeline to index the file content, but it wasn't scalable for me. Also, sometimes timeout or machine heap size causes problems. To support customers, we need a scalable solution to index millions of files' content without creating any problems. Please provide me your invaluable suggestions to get to the right solution. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 24, 2022, 8:41am UTC](https://discuss.elastic.co/t/how-to-solve-index-millions-of-files-content-with-scalable-approach/303084/2 "2022-04-24T08:41:28Z")
You could give a try to FSCrawler but it's not yet multithreaded. So you need to start one instance per subdir for example. This will change in the future. ---
**Author:** ![sushilprj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sushilprj/32/11593_2.png) [@sushilprj](https://discuss.elastic.co/u/sushilprj)\ **Post date:** [April 24, 2022, 9:14am UTC](https://discuss.elastic.co/t/how-to-solve-index-millions-of-files-content-with-scalable-approach/303084/3 "2022-04-24T09:14:29Z")
@dadoonet, Do you have any code sample or documentation focused on this problem? It will help me to create a quick POC around this. Thanks for your prompt response, and I appreciate your help and guidance. ---
**Author:** ![sushilprj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sushilprj/32/11593_2.png) [@sushilprj](https://discuss.elastic.co/u/sushilprj)\ **Post date:** [April 24, 2022, 10:16am UTC](https://discuss.elastic.co/t/how-to-solve-index-millions-of-files-content-with-scalable-approach/303084/4 "2022-04-24T10:16:13Z")
David, When you say one instance per subdir means one instance of Elasticsearch, right? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 24, 2022, 11:44am UTC](https://discuss.elastic.co/t/how-to-solve-index-millions-of-files-content-with-scalable-approach/303084/5 "2022-04-24T11:44:45Z")
No, one instance of FSCrawler per subdir. ---
**Author:** ![sushilprj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sushilprj/32/11593_2.png) [@sushilprj](https://discuss.elastic.co/u/sushilprj)\ **Post date:** [April 24, 2022, 11:49am UTC](https://discuss.elastic.co/t/how-to-solve-index-millions-of-files-content-with-scalable-approach/303084/6 "2022-04-24T11:49:02Z")
Thanks, @Christian_Dahlqvist ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 22, 2022, 11:49am UTC](https://discuss.elastic.co/t/how-to-solve-index-millions-of-files-content-with-scalable-approach/303084/7 "2022-05-22T11:49:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.