# FsCrawler does not do anything, does not index pfd's

**URL:** <https://discuss.elastic.co/t/fscrawler-does-not-do-anything-does-not-index-pfds/296747>\
**Category:** Elasticsearch\
**Created:** [February 9, 2022, 3:39pm UTC](https://discuss.elastic.co/t/fscrawler-does-not-do-anything-does-not-index-pfds/296747 "2022-02-09T15:39:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jank12](https://avatars.discourse-cdn.com/v4/letter/j/b5ac83/32.png) [@jank12](https://discuss.elastic.co/u/jank12)\
**Post date:** [February 9, 2022, 3:39pm UTC](https://discuss.elastic.co/t/fscrawler-does-not-do-anything-does-not-index-pfds/296747/1 "2022-02-09T15:39:08Z")

</div>

This n00b is thoroughly confused.

I have severa quite large pdf files I want to be able to full text search. They comrpise of raw text and images contaning text.

So I want to index the text in the pdf, OCR the images and index the result as well.

To this end I have setup an Ubuntu box with Elasticsearch and Kibana, and setup FSCrawler.

I get the json result from Elasticsearch and the kibaba website is functioning without issues.

I have put all PDF's in a folder, and created a fscrawler job to index them.

The job runs, but it appears to be doing nothing at all. It just sits there.

In kibana I can see an index, but it appears to be empty.

Here the fscrawler job:

```auto
---
name: "resumes"
fs:
  url: "/root/data/"
  update_rate: "1m"
  includes:
  - "*/*.pdf"
  json_support: true
  filename_as_id: true
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: true
  store_source: true
  index_content: true
  attributes_support: false
  raw_metadata: true
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: true
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: true
elasticsearch:
  nodes:
  - url: "http://192.168.1.126:8000"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: false

```

This is what I get when this job runs.. after some time, more of the same:

```auto
 [f.console] ,----------------------------------------------------------------------------------------------------.
| ,---,. .--.--. ,----.. ,--, 2.10-SNAPSHOT |
| ,' .' | / / '. / / \ ,--.'| |
| ,---.' || : /`. / | : : __,-. .---.| | :__ ,-. |
| | | .'; | |--` . | ;. /,' ,'/ /| /. ./|: : ' ,' ,'/ /| |
| : : : | : ;_ . ; /--` ' | |' | ,--.--. .-'-. ' || ' | ,---. ' | |' | |
| : | |-, \ \ `. ; | ; | | ,'/ \ /___/ \: |' | | / \ | | ,' |
| | : ;/| `----. \| : | ' : / .--. .-. | .-'.. ' ' .| | : / / |' : / |
| | | .' __\ \ |. | '___ | | ' \ __\/: . ./___ / \: '' : |__. ' / || | ' |
| ' : ' / /`--' /' ; : .'|; : | ," .--.; |. \ ' .\ | | '.'|' ; /|; : | |
| | | | '--'. / ' | '/ :| , ; / / ,. | \ \ ' \ |; : ;' | / || , ; |
| | : \ `--'---' | : / ---' ; : .' \ \ \ |--" | , / | : | ---' |
| | | ,' \ \ .' | , .-./ \ \ | ---`-' \ \ / |
| `----' `---` `--`---' '---" `----' |
+----------------------------------------------------------------------------------------------------+
| You know, for Files! |
| Made from France with Love |
| Source: https://github.com/dadoonet/fscrawler/ |
| Documentation: https://fscrawler.readthedocs.io/ |
`----------------------------------------------------------------------------------------------------'

15:37:07,840 INFO [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [34.3mb/949.3mb=3.62%], RAM [194mb/3.8gb=4.94%], Swap [0b/0b=0.0].
15:37:09,229 INFO [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
15:37:09,234 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
15:37:11,898 INFO [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.0
15:37:12,380 INFO [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.0
15:37:13,156 INFO [f.p.e.c.f.FsParserAbstract] FS crawler started for [resumes] for [/root/data/] every [1m]
15:37:13,318 WARN [o.e.c.RestClient] request [POST http://192.168.1.126:8000/resumes/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.0-bee86328705acaa9a6daede7140defd4d9ec56bd "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
15:37:13,365 WARN [o.e.c.RestClient] request [POST http://192.168.1.126:8000/resumes_folder/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.0-bee86328705acaa9a6daede7140defd4d9ec56bd "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

```

I cannot seam to get more verbose output. I would have expected some output like:

"found file input.pdf"  
"scanning an indexing text"  
"found image, applying ocr"  
....

or something like that. What am I doing wrong?

I used the latest versions of everything.

---

<div class="post-metadata">

**Author:** ![jank12](https://avatars.discourse-cdn.com/v4/letter/j/b5ac83/32.png) [@jank12](https://discuss.elastic.co/u/jank12)\
**Post date:** [February 9, 2022, 9:56pm UTC](https://discuss.elastic.co/t/fscrawler-does-not-do-anything-does-not-index-pfds/296747/2 "2022-02-09T21:56:00Z")

</div>

I have since also installed tesseract using apt (although the fscrawler download should contain it) and tika 2.2.1.

To no avail.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 10, 2022, 12:15am UTC](https://discuss.elastic.co/t/fscrawler-does-not-do-anything-does-not-index-pfds/296747/3 "2022-02-10T00:15:20Z")

</div>

If you have already run the job, that's expected as FSCrawler will only get the modified / added documents.

You can use the `--restart` option to make sure it starts again from scratch.

---

<div class="post-metadata">

**Author:** ![jank12](https://avatars.discourse-cdn.com/v4/letter/j/b5ac83/32.png) [@jank12](https://discuss.elastic.co/u/jank12)\
**Post date:** [February 10, 2022, 11:21pm UTC](https://discuss.elastic.co/t/fscrawler-does-not-do-anything-does-not-index-pfds/296747/4 "2022-02-10T23:21:16Z")

</div>

_facepalm self_

Thanks for your answer. This turned out to be the solution. I assumed since it hadn't actually indexed the documents yet, it would do that. It did not.

The --restart option fixed the issue. Should have found that myself.

In any case, thanks very much for taking the time to answer.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 10, 2022, 11:22pm UTC](https://discuss.elastic.co/t/fscrawler-does-not-do-anything-does-not-index-pfds/296747/5 "2022-03-10T23:22:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.