# FSCrawler path to remote dir **URL:** \ **Category:** Elasticsearch\ **Created:** [March 11, 2020, 9:20am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080 "2020-03-11T09:20:53Z")\ **Posts on this page:** 13\ **Page:** 2
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 2:02pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/21 "2020-03-11T14:02:52Z")
I think there is nothing to fix but add a better documentation... Could you try with: ```auto name: "dev_binary" fs: url: "/D:/TestData" server: hostname: "host" protocol: "ssh" username: "user" password: "pw" update_rate: "15m" excludes: - "*/~*" json_support: false filename_as_id: false add_filesize: true remove_deleted: true add_as_inner_object: false store_source: false index_content: true attributes_support: false raw_metadata: false xml_support: false index_folders: true lang_detect: false continue_on_error: false ocr: language: "eng" enabled: true pdf_strategy: "ocr_and_text" follow_symlinks: false elasticsearch: nodes: - url: "http://127.0.0.1:9200" bulk_size: 100 flush_interval: "5s" byte_size: "10mb" ``` ---
**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\ **Post date:** [March 11, 2020, 2:08pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/22 "2020-03-11T14:08:37Z")
Thank you really much! Should I update the documentation and submit a pull request? Working perfect now. ---
**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\ **Post date:** [March 11, 2020, 2:11pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/23 "2020-03-11T14:11:01Z")
But somehow if I change the update\_rate, it remains on 15 minutes after restart? Is my config in the correct format? Seems like update\_rate is assigned to server. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 2:12pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/24 "2020-03-11T14:12:15Z")
I'm on it 😉 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 2:13pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/25 "2020-03-11T14:13:04Z")
`update_rate` is just the pause duration between two internal runs once FSCrawler has started. ---
**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\ **Post date:** [March 11, 2020, 2:14pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/26 "2020-03-11T14:14:31Z")
Yes, that's what I thought too. After setting it to 1m, should it start every 1m? ``` 15:12:15,171 INFO [f.p.e.c.f.FsParserAbstract] FS crawler started for [dev_binary] for [/D:/TestData] every [15m] ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 2:16pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/27 "2020-03-11T14:16:42Z")
It will start a new run after it has finished the current run and with a delay of 1 minute. ---
**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\ **Post date:** [March 11, 2020, 2:17pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/28 "2020-03-11T14:17:28Z")
Aaaaah ok. Is there a way to set a schedule on when FSCrawler should look after new / edited files? I thought thats the way to go 🙂 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 2:28pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/29 "2020-03-11T14:28:04Z")
You can put FSCrawler in a crontab I guess and run it with the option `--loop 1`. It will exit after one run. See [https://fscrawler.readthedocs.io/en/latest/admin/cli-options.html#loop](https://fscrawler.readthedocs.io/en/latest/admin/cli-options.html#loop) ---
**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\ **Post date:** [March 12, 2020, 10:02am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/30 "2020-03-12T10:02:27Z")
Hey @dadoonet, I'm experiencing another issue with the `url` . If the `url` contains a subfolder, there is a exception raised. I got the following structure: - TestData | - TestDataSubFolder Exception: ``` 11:00:40,889 WARN [f.p.e.c.f.FsParserAbstract] Error while crawling /D:/TestData: String index out of range: -1 11:00:40,889 WARN [f.p.e.c.f.FsParserAbstract] Full stacktrace java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(Unknown Source) ~[?:1.8.0_202] at fr.pilato.elasticsearch.crawler.fs.FsParserAbstract.indexDirectory(FsParserAbstract.java:541) ~[fscrawler-core-2.7-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.FsParserAbstract.addFilesRecursively(FsParserAbstract.java:289) ~[fscrawler-core-2.7-SNAPSHOT.jar:?] at fr.pilato.elasticsearch.crawler.fs.FsParserAbstract.run(FsParserAbstract.java:149) [fscrawler-core-2.7-SNAPSHOT.jar:?] at java.lang.Thread.run(Unknown Source) [?:1.8.0_202] 11:00:40,905 DEBUG [f.p.e.c.f.FsParserAbstract] Fs crawler is going to sleep for 15m ``` It works if there isn't a subfolder. Quite strange Edit: Somehow the same exception is now also suddenly beeing raised if there isn't a subfolder > [@dadoonet](#): > > url: "/D:/TestData" doesn't work anymore... Im quite confused. Aber looking up the code and reproduce especially this line: ``` String rootdir = path.substring(0, path.lastIndexOf(File.separator)); ``` I always receive -1 for `path.lastIndexOf` which means the separator couldn't be find. So I designed my String like this: `\D:\TestData` and `lastIndexOf` delivers the correct index. Now im confused on how this worked before 😮 or if i had changed something which is now throwing errors... ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 12, 2020, 10:47am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/31 "2020-03-12T10:47:06Z")
Is that working now? If not, please open an issue. ---
**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\ **Post date:** [March 12, 2020, 10:51am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/32 "2020-03-12T10:51:18Z")
I copied a old version of `_status.json` in to the job directory and it's working now if there are no subfolder. Same exception get's raised if there are subfolders. Any Idea on that? I'll open a issue. EDIT: seems again like a problem with the `url`. Just tested it local on my windows machine and it all worked fine with subdirectorys. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 9, 2020, 10:51am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/33 "2020-04-09T10:51:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080.md?page=1)