# FSCrawler Question

**URL:** <https://discuss.elastic.co/t/fscrawler-question/75341>\
**Category:** Elasticsearch\
**Created:** [February 16, 2017, 1:56pm UTC](https://discuss.elastic.co/t/fscrawler-question/75341 "2017-02-16T13:56:28Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vikas\_Mestry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikas_mestry/32/42301_2.png) [@Vikas\_Mestry](https://discuss.elastic.co/u/Vikas_Mestry)\
**Post date:** [February 16, 2017, 1:56pm UTC](https://discuss.elastic.co/t/fscrawler-question/75341/1 "2017-02-16T13:56:28Z")

</div>

Hello,

I am trying to use fscrawler, would like to know if the following use case can be achieved using fscrawler.

1. I already have an index / type created with my custom mappings set
2. i am using the ingest-attachment plugin along with the ingest processor

Question : Can i use fscrawler to index pdf files into specified index/type/doc & specific field using the configurations / rest api ?

reason for doing this: i have very large documents which i would like to index, & the application i am using is on a windows ecosystem (using NEST client), getting a base64string out of large documents is giving me memory issues, so as an alternative would like to check if fscrawler can pull the documents directly & index them in specified index/type & against specific documentid (in specific field defined for attachment type)

Let me know if this all makes sense.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 16, 2017, 2:17pm UTC](https://discuss.elastic.co/t/fscrawler-question/75341/2 "2017-02-16T14:17:10Z")

</div>

> Can i use fscrawler to index pdf files into specified index/type/doc & specific field using the configurations / rest api ?

Yes. But FSCrawler gives less flexibility than ingest processors about field names. But the good news is that you can process a file with FSCrawler which will send it to elasticsearch through an ingest pipeline where you can simply rename a field to the desired target field you wish. See [GitHub - dadoonet/fscrawler: Elasticsearch File System Crawler (FS Crawler)](https://github.com/dadoonet/fscrawler#using-ingest-node-pipeline)

Is it what you are looking for?

---

<div class="post-metadata">

**Author:** ![Vikas\_Mestry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikas_mestry/32/42301_2.png) [@Vikas\_Mestry](https://discuss.elastic.co/u/Vikas_Mestry)\
**Post date:** [February 16, 2017, 2:47pm UTC](https://discuss.elastic.co/t/fscrawler-question/75341/3 "2017-02-16T14:47:43Z")

</div>

thanks @dadoonet for the quick response, i will try this out & update with my findings by tomorrow.

---

<div class="post-metadata">

**Author:** ![Vikas\_Mestry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikas_mestry/32/42301_2.png) [@Vikas\_Mestry](https://discuss.elastic.co/u/Vikas_Mestry)\
**Post date:** [February 17, 2017, 9:52am UTC](https://discuss.elastic.co/t/fscrawler-question/75341/4 "2017-02-17T09:52:16Z")

</div>

Hello @dadoonet,

I tried the above suggestion, but somehow the .bat doesn't do anything i.e. fscrawler doesn't start up at all I am using windows 7 & have an ES node running on my local system, following error comes up on the console if i attempt to terminate the batch job

```
D:\DevStuff\FsCrawler\bin>fscrawler --config_dir "D:\DevStuff\FsCrawler\attachtest_attach" attachtest_attach --loop 0 --rest
Exception in thread "main" java.util.NoSuchElementException
        at java.util.Scanner.throwFor(Scanner.java:862)
        at java.util.Scanner.next(Scanner.java:1371)
        at fr.pilato.elasticsearch.crawler.fs.FsCrawler.main(FsCrawler.java:212)

Terminate batch job (Y/N)? y

```

verified java version as follows

```
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

```

Following is the \_settings.json i have created,

```
{
  "name": "attachtest_attach",
  "fs": {
    "url": "D:\Files_Indexing\",
	"lang_detect" : true,
	"indexed_chars": "-1"
  },
  "elasticsearch": {
    "index": "attachtest",
    "type": "attach",
	"pipeline" : "attachment-pipeline",
	"nodes" : [
      { "host" : "127.0.0.1", "port" : 9200, "scheme" : "HTTP" }
    ]
  }
}

```

The above \_settings.json is placed in the following location, also specified the same against the --config\_dir param

```
D:\DevStuff\FsCrawler\attachtest_attach\_settings

```

Could you please let me know if i am missing something obvious.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 17, 2017, 10:12am UTC](https://discuss.elastic.co/t/fscrawler-question/75341/5 "2017-02-17T10:12:56Z")

</div>

Can you try with either:

- `--config_dir "D:\\DevStuff\\FsCrawler\\attachtest_attach"`
- `--config_dir "D:/DevStuff/FsCrawler/attachtest_attach"`

---

<div class="post-metadata">

**Author:** ![Vikas\_Mestry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikas_mestry/32/42301_2.png) [@Vikas\_Mestry](https://discuss.elastic.co/u/Vikas_Mestry)\
**Post date:** [February 17, 2017, 11:34am UTC](https://discuss.elastic.co/t/fscrawler-question/75341/6 "2017-02-17T11:34:37Z")

</div>

Tried the below, but no luck, the below errors are displayed when i terminate the bat (ctrl+c)

```
D:\DevStuff\FsCrawler\bin>fscrawler --config_dir "D:\\DevStuff\\FsCrawler\\attac
htest_attach" attachtest_attach --loop 0 --rest
Exception in thread "main" java.util.NoSuchElementException
        at java.util.Scanner.throwFor(Scanner.java:862)
        at java.util.Scanner.next(Scanner.java:1371)
Terminate batch job (Y/N)? y

D:\DevStuff\FsCrawler\bin>fscrawler --config_dir "D:/DevStuff/FsCrawler/attachte
st_attach" attachtest_attach --loop 0 --rest
Exception in thread "main" java.util.NoSuchElementException
        at java.util.Scanner.throwFor(Scanner.java:862)
        at java.util.Scanner.next(Scanner.java:1371)Terminate batch job (Y/N)? y

```

Is there any way we can generate logs for fscrawler

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 17, 2017, 6:51pm UTC](https://discuss.elastic.co/t/fscrawler-question/75341/7 "2017-02-17T18:51:47Z")

</div>

Was a bug on windows. Read [https://github.com/dadoonet/fscrawler/issues/320#issuecomment-280726439](https://github.com/dadoonet/fscrawler/issues/320#issuecomment-280726439)

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 17, 2017, 6:51pm UTC](https://discuss.elastic.co/t/fscrawler-question/75341/8 "2017-03-17T18:51:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.