# FSCrawler - Indexing mix of Big and small files - HTTP Entity too large error **URL:** \ **Category:** Elasticsearch\ **Created:** [January 12, 2024, 11:54am UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939 "2024-01-12T11:54:01Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![kamalsharma](https://avatars.discourse-cdn.com/v4/letter/k/f4b2a3/32.png) [@kamalsharma](https://discuss.elastic.co/u/kamalsharma)\ **Post date:** [January 12, 2024, 11:54am UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/1 "2024-01-12T11:54:01Z")
I have splitted a large text file into multiple files of 50 MB each. When the setting of bulk\_size is 1 in Fscrawler `_settings.json`, each file is indexed into Elasticsearch without any error. If the bulk size is increased I am getting the HTTP 413 Request Entity Too Large error. When I have a folder with small files and when the bulk\_size settings is 100, then also it is indexed perfectly. In some cases, I have a mix of large text files (50 MB each) and small files. If i keep the bulk\_size is 1. It is also indexing small files in a batch of 1 and the indexing time is increased. Is it possible to have a size range parameter so that the indexing time will be reduced. As of now we have "`ignore_above`" parameter so I can run file size below 20 MB (Example) with a "`bulk_size`" of 20. Then, if we have a range parameter I can again run the same folder with file size above 20 MB and upto 50 Mb with `bulk_size` of 1. Also please let me know is there any other way to solve this issue. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 12, 2024, 12:24pm UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/2 "2024-01-12T12:24:32Z")
Welcome! Could you share your full `_settings.json` file please? ---
**Author:** ![kamalsharma](https://avatars.discourse-cdn.com/v4/letter/k/f4b2a3/32.png) [@kamalsharma](https://discuss.elastic.co/u/kamalsharma)\ **Post date:** [January 12, 2024, 12:37pm UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/3 "2024-01-12T12:37:55Z")
Please find below the `_settings.json` file: ```auto { "name" : "fs-test-2024-001", "fs" : { "url" : "E:/Data/crawler_data/Test", "update_rate" : "12h", "includes" : [ "*/*.jpg", "*/*.jpeg", "*/*.png", "*/*.doc", "*/*.docx", "*/*.pdf", "*/*.txt", "*/*.sql" ], "excludes" : [ "*/*.zip", "*/*.rar", "*/*.exe", "*/*.mp4", "*/*.mp3" ], "json_support" : false, "xml_support" : false, "add_as_inner_object" : false, "filename_as_id" : false, "add_filesize" : true, "remove_deleted" : true, "store_source" : false, "index_content" : true, "attributes_support" : false, "raw_metadata" : false, "index_folders" : false, "lang_detect" : false, "continue_on_error" : false, "indexed_chars" : "-1", "ignore_above": "50mb", "checksum": "MD5", "ocr" : { "language" : "eng", "enabled" : true, "pdf_strategy": "ocr_and_text" } }, "elasticsearch" : { "nodes" : [ { "url": "https://localhost:9200" } ], "bulk_size": 1, "flush_interval": "5s" "username": "elastic", "password": "", "index": "fs-test-2024-001" }, "rest" : { "url": "http://127.0.0.1:8080/fscrawler" } } ``` If i change the bulk\_size: 4, i am getting the below error. ```auto 02:29:13,612 DEBUG [f.p.e.c.f.FsCrawlerImpl] Closing FS crawler [fs-dark-2024-001] 02:29:13,612 DEBUG [f.p.e.c.f.FsCrawlerImpl] FS crawler thread is now stopped 02:29:13,614 DEBUG [f.p.e.c.f.c.ElasticsearchClient] Closing Elasticsearch client manager 02:29:13,616 DEBUG [f.p.e.c.f.f.b.FsCrawlerBulkProcessor] Closing BulkProcessor 02:29:13,617 DEBUG [f.p.e.c.f.f.b.FsCrawlerBulkProcessor] BulkProcessor is now closed 02:29:13,620 DEBUG [f.p.e.c.f.s.FsCrawlerManagementServiceElasticsearchImpl] Elasticsearch Management Service stopped 02:29:13,620 DEBUG [f.p.e.c.f.c.ElasticsearchClient] Closing Elasticsearch client manager 02:29:13,620 DEBUG [f.p.e.c.f.f.b.FsCrawlerBulkProcessor] Closing BulkProcessor 02:29:13,621 DEBUG [f.p.e.c.f.f.b.FsCrawlerBulkProcessor] BulkProcessor is now closed 02:29:13,621 DEBUG [f.p.e.c.f.f.b.FsCrawlerBulkProcessor] Executing [4] remaining actions 02:29:13,622 DEBUG [f.p.e.c.f.f.b.FsCrawlerSimpleBulkProcessorListener] Going to execute new bulk composed of 4 actions 02:29:16,568 DEBUG [f.p.e.c.f.c.ElasticsearchEngine] Sending a bulk request of [4] documents to the Elasticsearch service 02:29:16,735 DEBUG [f.p.e.c.f.c.ElasticsearchClient] bulk a ndjson of 230246338 characters 02:29:17,815 DEBUG [f.p.e.c.f.c.ElasticsearchClient] Error while running POST https://localhost:9200/_bulk: 02:29:17,815 WARN [f.p.e.c.f.f.b.FsCrawlerSimpleBulkProcessorListener] Error executing bulk jakarta.ws.rs.ClientErrorException: HTTP 413 Request Entity Too Large at org.glassfish.jersey.client.JerseyInvocation.createExceptionForFamily(JerseyInvocation.java:985) ~[jersey-client-3.1.5.jar:?] at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:967) ~[jersey-client-3.1.5.jar:?] at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:755) ~[jersey-client-3.1.5.jar:?] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) ~[jersey-client-3.1.5.jar:?] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-3.1.5.jar:?] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-3.1.5.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.1.5.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.1.5.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.1.5.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-3.1.5.jar:?] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-3.1.5.jar:?] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) ~[jersey-client-3.1.5.jar:?] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:450) ~[jersey-client-3.1.5.jar:?] at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:871) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpPost(ElasticsearchClient.java:847) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.bulk(ElasticsearchClient.java:808) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchEngine.bulk(ElasticsearchEngine.java:82) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchEngine.bulk(ElasticsearchEngine.java:31) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.framework.bulk.FsCrawlerBulkProcessor.execute(FsCrawlerBulkProcessor.java:146) [fscrawler-framework-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.framework.bulk.FsCrawlerBulkProcessor.internalClose(FsCrawlerBulkProcessor.java:101) [fscrawler-framework-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.framework.bulk.FsCrawlerBulkProcessor.close(FsCrawlerBulkProcessor.java:77) [fscrawler-framework-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.close(ElasticsearchClient.java:452) [fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerDocumentServiceElasticsearchImpl.close(FsCrawlerDocumentServiceElasticsearchImpl.java:60) [fscrawler-core-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.close(FsCrawlerImpl.java:170) [fscrawler-core-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.runner(FsCrawlerCli.java:399) [fscrawler-cli-2.10-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:119) [fscrawler-cli-2.10-SNAPSHOT.jar:?] 02:29:17,832 DEBUG [f.p.e.c.f.s.FsCrawlerDocumentServiceElasticsearchImpl] Elasticsearch Document Service stopped ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 12, 2024, 12:44pm UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/4 "2024-01-12T12:44:50Z")
The problem is here: ``` "indexed_chars" : "-1" ``` You are asking to extract the whole text content. Not sure how much it represents actually but that's may be too much anyway. There's a limit on Elasticsearch side, which is by default 100mb. And I would not recommend increasing this limit unless you know exactly what you are doing. Instead of using `bulk_size`, you could use `byte_size`: ``` elasticsearch.byte_size: 80mb ``` See [Elasticsearch settings — FSCrawler 2.10-SNAPSHOT documentation](https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#bulk-settings) HTH ---
**Author:** ![kamalsharma](https://avatars.discourse-cdn.com/v4/letter/k/f4b2a3/32.png) [@kamalsharma](https://discuss.elastic.co/u/kamalsharma)\ **Post date:** [January 12, 2024, 12:50pm UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/5 "2024-01-12T12:50:48Z")
Yes, I am aware of the limit of 100 MB on elasticsearch side. It is a big text file and it is splitted into multiple files of 50 MB. I have also tested with byte\_size of 60 MB and removed the bulk\_size. In this case also i am getting the error. As i have mentioned earlier, if bulk\_size is 1, it is successful ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 12, 2024, 12:52pm UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/6 "2024-01-12T12:52:01Z")
So I'd suggest that you split the text in smaller pieces, like 10mb each. ---
**Author:** ![kamalsharma](https://avatars.discourse-cdn.com/v4/letter/k/f4b2a3/32.png) [@kamalsharma](https://discuss.elastic.co/u/kamalsharma)\ **Post date:** [January 12, 2024, 12:53pm UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/7 "2024-01-12T12:53:48Z")
I have a query, if the byte\_size is 60 MB and we didn't mention bulk\_size, how much document should be indexed at a time ---
**Author:** ![kamalsharma](https://avatars.discourse-cdn.com/v4/letter/k/f4b2a3/32.png) [@kamalsharma](https://discuss.elastic.co/u/kamalsharma)\ **Post date:** [January 22, 2024, 1:54pm UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/8 "2024-01-22T13:54:47Z")
Hi David, There are 2 settings bulk\_size and byte\_size. If I keep bulk\_size to say 50 and the byte\_size as 80mb. While adding the files for Bulk Index, is it possible for us to check whether byte\_size limit is crossed. If it is crossed after 35th file, is it possible for us to bulk index the 35 files alone and the remaining files will be indexed in the next batch. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 31, 2024, 5:12pm UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/9 "2024-01-31T17:12:19Z")
> [@kamalsharma](#): > > is it possible for us to check whether byte\_size limit is crossed I think there's an issue. The `byte_size` limit is totally ignored... Could you open an issue? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 28, 2024, 5:12pm UTC](https://discuss.elastic.co/t/fscrawler-indexing-mix-of-big-and-small-files-http-entity-too-large-error/350939/10 "2024-02-28T17:12:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.