# FSCrawler failure handling **URL:** \ **Category:** Elasticsearch\ **Created:** [September 20, 2021, 11:13am UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615 "2021-09-20T11:13:41Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Francois\_Saab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francois_saab/32/91763_2.png) [@Francois\_Saab](https://discuss.elastic.co/u/Francois_Saab)\ **Post date:** [September 20, 2021, 11:13am UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/1 "2021-09-20T11:13:41Z")
Hello, We are considering the question: how FSCrawler manages failure? What happened if FSCrawler failed to upload a certain file because of a failure on the ElasticSearch side? Does FSCrawler implements an automatic failure handling mechanism? Consider the scenario, FSCrawler fails to upload a file because Elasticsearch is Down. 1. Can we have FSCrawler to retry uploading the file? can we configure number\_retries? 2. Can we have FSCrawler to log the following: a. File on which failure happened b. Exception information What is the structure (i.e. fields) of the log file "documents.log"? I looked at that file but i didn't find reference to the "file name" on which the failure happened. Thank you ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 11:54am UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/2 "2021-09-20T11:54:01Z")
Hello François > [@Francois\_Saab](#): > > What happened if FSCrawler failed to upload a certain file because of a failure on the Elasticsearch side? It will just log it to the logs and/or to the console. I'm not sure how `fs.continue-on-error` setting will play a role here. ([https://fscrawler.readthedocs.io/en/fscrawler-2.7/admin/fs/local-fs.html#continue-on-error](https://fscrawler.readthedocs.io/en/fscrawler-2.7/admin/fs/local-fs.html#continue-on-error)). I did not look at the code but I assume that this setting only apply to the "input" part and not the "output" part. > 1. Can we have FSCrawler to retry uploading the file? can we configure number\_retries? No and no. That's something that should be implemented. A Dead Letter Queue (DLQ) mechanism would be great to have. > 1. Can we have FSCrawler to log the following: > a. File on which failure happened > b. Exception information See my comments later. I think the exception is provided. it's not the case? > [@Francois\_Saab](#): > > What is the structure (i.e. fields) of the log file "documents.log"? I looked at that file but i didn't find reference to the "file name" on which the failure happened. The code is: ```auto public static void documentError(String id, String path, String error) { documentLogger.error("[{}][{}] {}", id, path, error); } ``` But when the document is sent to elasticsearch, the `path` value is "unknown" within the bulk request. I think that could be a great thing to add. Would you like to open a feature request? If you activate the debug mode for documents, you could see what is the id of the document with the full path and then when you have an error, you could try to find the same id in the debug part... Not ideal for sure. ---
**Author:** ![Francois\_Saab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francois_saab/32/91763_2.png) [@Francois\_Saab](https://discuss.elastic.co/u/Francois_Saab)\ **Post date:** [September 20, 2021, 12:37pm UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/3 "2021-09-20T12:37:19Z")
I created this new feature request: > > > \- Target feature: > > Provide information for the physical path of the file for …which FSCralwer has failed to operate on (e.g. index in ES). > > \- Current Situation: > > Currently you are logging as the following example shows in documents.log: > > \`2021-07-01 22:11:53,627 \[ERROR\] \[375040c5d4baa5408ae296233dc6e79c\]\[null\] ElasticsearchException\[Elasticsearch exception \[type=cluster\_block\_exception, reason=index \[data\_files\] blocked by: \[TOO\_MANY\_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block\];\]\]\` > > \- Issue with current implementation: > > No information on the physical path of the file. And therefore we are not able to locate the file in question to retry uploading it. ---
**Author:** ![Francois\_Saab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francois_saab/32/91763_2.png) [@Francois\_Saab](https://discuss.elastic.co/u/Francois_Saab)\ **Post date:** [September 20, 2021, 12:46pm UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/4 "2021-09-20T12:46:26Z")
Hello David, thanks for your Quick reply. Below is an example of documents log: `2021-07-01 22:11:53,627 [ERROR] [375040c5d4baa5408ae296233dc6e79c][null] ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [data_files] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]` A per my understanding of your explanation. It logs mainly 3 things: `375040c5d4baa5408ae296233dc6e79c <-- Id` `null: --> path to be added` `Elasticsearch exception [type=cluster_block_exception, reason=index [data_files] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];] --> the exception` I am supposing that the Id field is what FSCrawler assigns for the elastic search document. true? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 1:22pm UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/5 "2021-09-20T13:22:41Z")
> [@Francois\_Saab](#): > > I created this new feature request: Great. thanks! > [@Francois\_Saab](#): > > I am supposing that the Id field is what FSCrawler assigns for the Elasticsearch document. true? That's true. It's using the full path to generate an id. ``` Elasticsearch exception [type=cluster_block_exception, reason=index [data_files] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]; ``` About this error, did you intentionally changed the watermark settings to make FSCrawler failing? For tests purpose I mean. ---
**Author:** ![Francois\_Saab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francois_saab/32/91763_2.png) [@Francois\_Saab](https://discuss.elastic.co/u/Francois_Saab)\ **Post date:** [September 20, 2021, 1:40pm UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/6 "2021-09-20T13:40:12Z")
Thanks. No this failure happened without any intervention from my side. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 2:05pm UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/7 "2021-09-20T14:05:28Z")
Ok. So that's something you really need to fix then! 🙂 Make sure you have enough disk space on your machine. Specifically on a production machine. ---
**Author:** ![Francois\_Saab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francois_saab/32/91763_2.png) [@Francois\_Saab](https://discuss.elastic.co/u/Francois_Saab)\ **Post date:** [September 20, 2021, 4:59pm UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/8 "2021-09-20T16:59:09Z")
Yes i will. thanks for mentioning this. this was from my development machine which is full of databases 🙂 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 6:16pm UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/9 "2021-09-20T18:16:15Z")
On dev machine, you can change the watermarks threshold or disable the beahvior by setting `cluster.routing.allocation.disk.threshold_enabled` to `false`. See [Cluster-level shard allocation and routing settings | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation) for more info. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 18, 2021, 6:16pm UTC](https://discuss.elastic.co/t/fscrawler-failure-handling/284615/10 "2021-10-18T18:16:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.