# FScrawler stuck at 2.6gb index size **URL:** \ **Category:** Elasticsearch\ **Created:** [December 9, 2019, 10:53am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094 "2019-12-09T10:53:26Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [December 9, 2019, 10:53am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/1 "2019-12-09T10:53:26Z")
I have elasticsearch that uses fscrawler to index a folder that is about 5TB. The \_settings.yml has not been modified except that it should update after every 3 hours. ![rename](https://us1.discourse-cdn.com/elastic/original/3X/c/2/c278790fa749d94bf57881e390a8532abf7b0226.png) Everything goes well until the index grew to 2.6gb then the index stalled. What could be the problem? Is it something to do with the memory? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 10, 2019, 12:23am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/2 "2019-12-10T00:23:46Z")
How many documents have been indexed? How many are missing? Anything in FSCrawler logs? ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [December 10, 2019, 9:06am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/3 "2019-12-10T09:06:27Z")
Around 800,000 out of about 3,000,00 documents indexed so far.. My server's memory is 40GB and I have allocated about 12GB for elasticsearch. I am running fscrawler on the terminal but then after 2-3 days when I query index details, it appeared to have stalled though there is no error at the terminal. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 10, 2019, 10:54am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/4 "2019-12-10T10:54:00Z")
How much heap did you give to FSCrawler? ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [December 10, 2019, 8:25pm UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/5 "2019-12-10T20:25:32Z")
I am sorry I don't know how to assign heap to FSCrawler. I just used the default settings. Kindly advice how I can do that. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 10, 2019, 9:00pm UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/6 "2019-12-10T21:00:34Z")
Here you go: [https://fscrawler.readthedocs.io/en/latest/admin/jvm-settings.html](https://fscrawler.readthedocs.io/en/latest/admin/jvm-settings.html) ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [December 11, 2019, 5:23am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/7 "2019-12-11T05:23:46Z")
Got it! Thank you. 3 quick ones, what is the default FSCrawler heap size? how much heap should be sufficient for such amount of data? and maybe just for clarification, since I have my update\_rate at 3hrs, does that mean it will start updating even before it completes indexing? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 11, 2019, 5:52am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/8 "2019-12-11T05:52:59Z")
1. I don't know TBH as I don't think I set it. So it's probably 256m to 1g. 2. As you have a lot of memory, I'd try with 4 gb. 3. no. 3h is defined after the end of a run. ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [December 11, 2019, 8:51am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/9 "2019-12-11T08:51:38Z")
Trying to set up heap gives error --\> Invalid maximum heap size ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/6/e/6ed4396db0c214ae27c2400c498c053d0163ba03.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 11, 2019, 10:16am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/10 "2019-12-11T10:16:57Z")
You don't have enough free memory available on your machine apparently. ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [December 11, 2019, 10:18am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/11 "2019-12-11T10:18:55Z")
.. Do you think I am doing it right? because it gives the same error even with 512mb. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 11, 2019, 12:56pm UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/12 "2019-12-11T12:56:16Z")
May be you should define a system var using the control panel and restart the console? ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [December 11, 2019, 12:59pm UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/13 "2019-12-11T12:59:49Z")
Do you mean like this? ![Capture2](https://us1.discourse-cdn.com/elastic/original/3X/1/7/17de031794d28b9a6c3110d5ec3199d18e963052.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 11, 2019, 1:15pm UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/14 "2019-12-11T13:15:09Z")
Yes but not with the double quotes and not with bin/fscrawler ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [December 12, 2019, 10:30am UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/15 "2019-12-12T10:30:16Z")
So I used the system variable approach and started indexing afresh. I believe that HEAP as shown below belongs to FSCrawler. As you can see, a lot of the memory seems unused.. from the ram to the heap. ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/9/4/946c19823fbe8463135a72a8328b1c917df9fecf.png) I don't know how long it might take to index my 5TB, data have you encountered such amount of data before? How long did it take to index? How do I know if the indexing is still going on or it has hang? because last time there was no progress for more than two days (after about 350GB data indexing, index size was 2.6gb). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 12, 2019, 1:18pm UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/16 "2019-12-12T13:18:07Z")
> [@Denn](#): > > have you encountered such amount of data before? No. Never tried with lot of data. I know that some users are also indexing a lot. > [@Denn](#): > > How do I know if the indexing is still going on or it has hang? Well. That's a good question. I should add a REST endpoint for it so you can know. Or something in the console writing that work is in progress but without needing `--debug` option. Would you like to open an issue? ---
**Author:** ![Denn](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@Denn](https://discuss.elastic.co/u/Denn)\ **Post date:** [December 12, 2019, 1:30pm UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/17 "2019-12-12T13:30:04Z")
> [@](#): > > Would you like to open an issue? Yeah sure. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 9, 2020, 1:30pm UTC](https://discuss.elastic.co/t/fscrawler-stuck-at-2-6gb-index-size/211094/18 "2020-01-09T13:30:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.