# Elasticsearch considerations for ingesting large files **URL:** \ **Category:** Elasticsearch\ **Created:** [April 10, 2020, 9:22am UTC](https://discuss.elastic.co/t/elasticsearch-considerations-for-ingesting-large-files/227464 "2020-04-10T09:22:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Post date:** [April 10, 2020, 9:22am UTC](https://discuss.elastic.co/t/elasticsearch-considerations-for-ingesting-large-files/227464/1 "2020-04-10T09:22:53Z")
Hello Team, Just wanted to understand the limits , scaling and performance of Elasticsearch, what should be the considerations while ingesting large files (40-50) GB, metadata and making it searchable. For examples microsoft office documents , Images , zip files with more than 30-50 GB size. what is the best way to get near real time search experience. Thanks for your help and support Thanks and Regards, Aditya Deshpande ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 10, 2020, 9:37am UTC](https://discuss.elastic.co/t/elasticsearch-considerations-for-ingesting-large-files/227464/2 "2020-04-10T09:37:41Z")
Elasticsearch is not designed to store big binary blobs. If you can extract the text and index just the text though that's another story. For ZIP files, that's another story. I'd not try to index the content of a whole zip but I'd more index individual files within the ZIP instead. So unzip, index each file. You can have a look at [FSCrawler project](https://fscrawler.readthedocs.io/en/latest/) BTW which can help you started with binary documents. If your documents are stored in Sharepoint, Dropbox, ... you should look at [Workplace Search](https://www.elastic.co/workplace-search). ---
**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Post date:** [April 10, 2020, 10:10am UTC](https://discuss.elastic.co/t/elasticsearch-considerations-for-ingesting-large-files/227464/3 "2020-04-10T10:10:16Z")
Thank you very much dadoonet, I will try to explore [Workplace Search](https://www.elastic.co/workplace-search). could you please any technical documents available for this. will it work if my documenst are on harddrive. meanwhile could you please help me understand if i want to extract the text and index it into elasticsearch lets say using fscrawler and ocr how can i achieve near real time search experience. parsing big files and extracting text would be time / memory and cpu consuming tasks. is there any way we can do it efficiently. Thank you for your help and support. Thank you, Aditya ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 10, 2020, 10:24am UTC](https://discuss.elastic.co/t/elasticsearch-considerations-for-ingesting-large-files/227464/4 "2020-04-10T10:24:13Z")
> [@adityaPsl](#): > > will it work if my documenst are on harddrive. Not yet. But I'm planning to connect FSCrawler with it. > > > Let's connect to as well to https://swiftype.com/documentation/enterprise-search…/api/custom-sources > [@adityaPsl](#): > > meanwhile could you please help me understand if i want to extract the text and index it into elasticsearch lets say using fscrawler and ocr how can i achieve near real time search experience. parsing big files and extracting text would be time / memory and cpu consuming tasks. is there any way we can do it efficiently. Well. Just start from [Tutorial — FSCrawler 2.10-SNAPSHOT documentation](https://fscrawler.readthedocs.io/en/latest/user/tutorial.html) ---
**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Post date:** [April 11, 2020, 2:42am UTC](https://discuss.elastic.co/t/elasticsearch-considerations-for-ingesting-large-files/227464/5 "2020-04-11T02:42:35Z")
Thank you very much dadoonet, I think fscrawler should be sufficient to crawl contents on s3 bucket. few queries FS Crawler: 1. is there any limit (per file) on sending crawled data from fscrawler to elasticsearch, since this transfer would happen over http/s. 2. Does FSCrawler supports parallel processing ? If multiples files are getting uploaded to the drive will it try to crawl all with mutiple threads or synchronously one after another. 3. is it a good idea storing huge data on elasticsearch , i assume 40 GB file parsed and indexed in elasticsearch would approximately occupy the same sapce. Thank you, Aditya ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 11, 2020, 9:31am UTC](https://discuss.elastic.co/t/elasticsearch-considerations-for-ingesting-large-files/227464/6 "2020-04-11T09:31:33Z")
> [@adityaPsl](#): > > is there any limit (per file) on sending crawled data from fscrawler to elasticsearch, since this transfer would happen over http/s. Yes. See [Elasticsearch settings — FSCrawler 2.10-SNAPSHOT documentation](https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#bulk-settings) > [@adityaPsl](#): > > Does FSCrawler supports parallel processing ? No. See > > > Since piping graphical PDFs without actual text through tesseract is very time i…ntensive, support for pareallel crawling would be a great. This could be done in two ways: > > \- via a multi-threaded process model > \- by having two distinct status codes for a document in the index > > The latter should be relatively simple to do and it would enable the operator to just run as many fscrawlers on one particular directory as wanted. Maybe worth to take a look at. But you can run multiple instances of FSCrawler in parallel to monitor different sub dirs. > [@adityaPsl](#): > > is it a good idea storing huge data on elasticsearch No. I don't think it is. You should only store the extracted text or a part of the extracted text. By default, FSCrawler extracts only 10000 characters. See [Local FS settings — FSCrawler 2.10-SNAPSHOT documentation](https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html#extracted-characters) ---
**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Post date:** [April 11, 2020, 11:33am UTC](https://discuss.elastic.co/t/elasticsearch-considerations-for-ingesting-large-files/227464/7 "2020-04-11T11:33:37Z")
Thank you very much dadoonet. It really helps 🙂 Thank you, Aditya ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 9, 2020, 11:33am UTC](https://discuss.elastic.co/t/elasticsearch-considerations-for-ingesting-large-files/227464/8 "2020-05-09T11:33:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.