# Why is my index not growing past 4GB when using fscrawler? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2020, 1:54pm UTC](https://discuss.elastic.co/t/why-is-my-index-not-growing-past-4gb-when-using-fscrawler/232838 "2020-05-15T13:54:36Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [May 15, 2020, 1:54pm UTC](https://discuss.elastic.co/t/why-is-my-index-not-growing-past-4gb-when-using-fscrawler/232838/1 "2020-05-15T13:54:36Z")
So I am trying to index about 2.5TB of data using fscrawler that is about 3 million files.. I have 40GB of ram of which I have set aside 20GB heap for fscrawler for maximum throughput. ```auto C:\Elastic\fscrawler-MAR15\bin>fscrawler trial2 14:58:51,919 INFO [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [18.8gb/19.1gb=98.43%], RAM [37gb/40.9gb=90.53%], Swap [1.8gb/47.1gb=3.97%]. 14:58:52,998 INFO [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.1.1 14:58:53,185 INFO [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler 14:58:53,185 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C. ``` However, only about 1 million of data has been indexed so far with the index size remaining at 4GB for the last 3 weeks. I don't know if indexing is going on or it has stalled. (kindly also explain to me what the swap memory is with regard to fscrawler, does mine which is 1.8gb affect the performance?) N/B I once had to restart indexing because I found an error "your computer is low on memory.. save files and close programs. Java (TM) Platform binary". kindly advice me on the indexing situation and memory. thank you. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 15, 2020, 2:13pm UTC](https://discuss.elastic.co/t/why-is-my-index-not-growing-past-4gb-when-using-fscrawler/232838/2 "2020-05-15T14:13:46Z")
Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable. Instead, paste the text and format it with `` icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer. It would be great if you could update your post to solve this. > kindly also explain to me what the swap memory is It's when you don't have enough RAM. OS can use the hard disk as a memory. But this makes everything super slow. > [@Denn](#): > > So I am trying to index about 2.5TB of data using fscrawler The important number to look at is the number of documents that needs to be indexed. You can't really compare the source size and the size in elasticsearch as only the extracted content is indexed. So I can see that almost 1m documents have been indexed by FSCrawler. Do you know how many documents should have been indexed? FSCrawler visited somehow 100k folders. Sadly, there is no "progress report" available yet in FSCrawler. > > > So it is a great feature that FSCrawler keeps logs. But as issue #834 describes,… there is need to monitor progress perhaps information like: > > \*current indexing folder. > \*status of whether the process is going on well or it has freeze. > > This can greatly assist in debugging other issues. The only way to know is by starting FSCrawler with the `--debug` option. ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [May 15, 2020, 2:45pm UTC](https://discuss.elastic.co/t/why-is-my-index-not-growing-past-4gb-when-using-fscrawler/232838/3 "2020-05-15T14:45:13Z")
I have tried to update the question. Immediately after posting this question, I got another memory full error.. What could be causing this? Because from the task manager, memory used is at about 30% . Is my heap and ram sufficient? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 15, 2020, 3:07pm UTC](https://discuss.elastic.co/t/why-is-my-index-not-growing-past-4gb-when-using-fscrawler/232838/4 "2020-05-15T15:07:11Z")
> [@Denn](#): > > I got another memory full error Could you share it? What are the fscrawler job settings? ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [May 15, 2020, 5:50pm UTC](https://discuss.elastic.co/t/why-is-my-index-not-growing-past-4gb-when-using-fscrawler/232838/5 "2020-05-15T17:50:49Z")
I am sorry but I can only use a photo here. This is the error ! [tymHb|367x289](https://discuss.elastic.co/uploads/short-url/zFc4OMPGbwInZabucy8qou3MrDv.png) My job settings looks like this: ```auto --- name: "index" fs: url: "\\\\DESKTOP-O5VVOG6\\shared_docs" update_rate: "15m" excludes: - "*/~*" json_support: false filename_as_id: false add_filesize: true remove_deleted: true add_as_inner_object: false store_source: false index_content: true attributes_support: false raw_metadata: false xml_support: false index_folders: true lang_detect: false continue_on_error: true ocr: language: "eng" enabled: true pdf_strategy: "ocr_and_text" elasticsearch: nodes: - url: "http://127.0.0.1:9200" bulk_size: 100 flush_interval: "5s" byte_size: "10mb" ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 18, 2020, 9:58am UTC](https://discuss.elastic.co/t/why-is-my-index-not-growing-past-4gb-when-using-fscrawler/232838/6 "2020-05-18T09:58:32Z")
That's strange that Windows complains about the memory usage. I'd expect an `OutOfMemory` java exception instead. The memory is supposed to be allocated and available for the process. Is there any option on windows machines to make sure that a process can actually lock the memory? ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [May 18, 2020, 5:43pm UTC](https://discuss.elastic.co/t/why-is-my-index-not-growing-past-4gb-when-using-fscrawler/232838/7 "2020-05-18T17:43:59Z")
So I think I found the error.. my c drive was full (location of my index).. after reboot I found 0 bytes available.. so I added more memory and tried again... Am monitoring it and it seem to be running fine.. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 15, 2020, 5:44pm UTC](https://discuss.elastic.co/t/why-is-my-index-not-growing-past-4gb-when-using-fscrawler/232838/8 "2020-06-15T17:44:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.