# FSCrawler path to remote dir

**URL:** <https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080>\
**Category:** Elasticsearch\
**Created:** [March 11, 2020, 9:20am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080 "2020-03-11T09:20:53Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 9:20am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/1 "2020-03-11T09:20:53Z")

</div>

I'm using FSCrawler by @dadoonet and im trying to index files on a remote server.  
The ssh connection is established, but it can't find the path. I tried all variants I could think of in the URL String.

My config looks like this:

```
name: "dev_binary"
fs:
  url: "D:\\TestData"
server:
  hostname: "host"
  protocol: "ssh"
  username: "user"
  password: "pw"
  update_rate: "15m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "http://127.0.0.1:9200"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"

```

Any idea or example on how to set the path properly?

I'm on a Windows machine.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 10:37am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/2 "2020-03-11T10:37:51Z")

</div>

I believe that URL should be something like `url: "/TestData"`.

To find the right URL, you should use a SSH client and run something like:

```auto
ssh host

```

Once connected, try to find the right dir.

```auto
cd /
ls

```

Or

```auto
cd /TestData

```

Once you have the right dir, use that name in the `url` field.

---

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 10:44am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/3 "2020-03-11T10:44:15Z")

</div>

Could it be a problem, that when I connect I'm in

```
C:\User\user

```

And I have to use Data from `D:`?

I looked up if the Folder exists and i can `cd` to it:

```
user@host C:\Users\user>cd /

user@host C:\>D:

user@host D:\>cd TestData

user@host D:\TestData>

```

> I believe that URL should be something like `url: "/TestData"` .

It doesn't work when I use this. Note that I'm on a Windows Server...

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 11:03am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/4 "2020-03-11T11:03:01Z")

</div>

Could you try to run (using ssh):

```auto
cd d:/TestData
ls

```

---

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 11:06am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/5 "2020-03-11T11:06:33Z")

</div>

Sure, the result:

```
user@host C:\Users\user>cd d:/TestData

user@host C:\Users\user>cd D:/TestData

```

It won't switch the directory. I used PuTTy as SSH client.  
Am I missing something?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 11:08am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/6 "2020-03-11T11:08:31Z")

</div>

> [@LaaKii](#):
>
> Am I missing something?

No. I believe this is an issue with Windows filesystem. Let me think about it and come back to you.

---

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 11:09am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/7 "2020-03-11T11:09:36Z")

</div>

That's what I thought as well since you have to switch directories by just using the drive name.

Thank you really much for your effort.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 11:09am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/8 "2020-03-11T11:09:42Z")

</div>

Could you try:

```auto
cd /d d:/TestData

```

---

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 11:10am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/9 "2020-03-11T11:10:24Z")

</div>

> cd /d d:/TestData

That worked! I'm now in the correct directory

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 11:11am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/10 "2020-03-11T11:11:45Z")

</div>

So I need to look if I can use that switch in FSCrawler. Let me check.

You tried already to use `"url": "d:/TestData"`, right?  
If not, could you try it.

---

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 11:14am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/11 "2020-03-11T11:14:04Z")

</div>

Yes, I just rerun it:

```
12:14:17,579 DEBUG [f.p.e.c.f.c.s.FileAbstractorSSH] Opening SSH connection to xxx
12:14:21,954 DEBUG [f.p.e.c.f.c.s.FileAbstractorSSH] SSH connection successful
    12:13:03,000 WARN [f.p.e.c.f.FsParserAbstract] Error while crawling d:/TestData: d:/TestData doesn't exists.
    12:13:03,000 WARN [f.p.e.c.f.FsParserAbstract] Full stacktrace
    java.lang.RuntimeException: d:/TestData doesn't exists.

```

Does it matter, that `url` isn't in quotes in my config?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 11:16am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/12 "2020-03-11T11:16:02Z")

</div>

> [@LaaKii](#):
>
> Does it matter, that `url` isn't in quotes in my config?

No. Sorry. My bad. I was still thinking it was json but I switched the settings to yaml.

Last question (may be 🙂 ). Could you run:

```auto
ls d:/TestData

```

---

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 11:18am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/13 "2020-03-11T11:18:34Z")

</div>

> Last question (may be 🙂 ). Could you run:
> 
> ```auto
> ls d:/TestData
> 
> ```

No problems in asking, I'm happy you help me with my problem 🙂

I can't execute `ls` commands in PuTTY, since I'm on a windows machine 😕

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 11:20am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/14 "2020-03-11T11:20:33Z")

</div>

I meant. Can you ssh to the machine and run that command from the SSH Client?

---

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 11:22am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/15 "2020-03-11T11:22:16Z")

</div>

I did that, but I can't execute `ls` in the SSH Client:

```
user@host C:\Users\user>ls d:/TestData
'ls' is not recognized as an internal or external command,
operable program or batch file.

user@host C:\Users\user>

```

Oh maybe to clarify: The remote server is also a windows machine.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 11:29am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/16 "2020-03-11T11:29:42Z")

</div>

> Oh maybe to clarify: The remote server is also a windows machine.

Yeah. I got that.  
Sounds like I need to start a windows VM and add a patch to the code.

Could you open an issue in [Issues · dadoonet/fscrawler · GitHub](https://github.com/dadoonet/fscrawler/issues)?

---

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 11:30am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/17 "2020-03-11T11:30:36Z")

</div>

> Could you open an issue in [Issues · dadoonet/fscrawler · GitHub](https://github.com/dadoonet/fscrawler/issues)

Feature or Bug ? Sure I can 🙂

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 11:56am UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/18 "2020-03-11T11:56:31Z")

</div>

It's a bug to me.

---

<div class="post-metadata">

**Author:** ![LaaKii](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LaaKii](https://discuss.elastic.co/u/LaaKii)\
**Post date:** [March 11, 2020, 12:03pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/19 "2020-03-11T12:03:56Z")

</div>

I opened it. If you want me to add extra details to the issue I surely can. I mainly linked to this post here.

Can you estimate how much effort it will be to change this, or even when it will work?  
Thank you so much 🙂

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 11, 2020, 12:23pm UTC](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080/20 "2020-03-11T12:23:40Z")

</div>

I don't know. I first need to have a VM running with an ssh server on it. WIP.  
But I'm on it. 😉

[Next page](https://discuss.elastic.co/t/fscrawler-path-to-remote-dir/223080.md?page=2)