# FSCrawler Rest with Elasticsearch

**URL:** <https://discuss.elastic.co/t/fscrawler-rest-with-elasticsearch/323803>\
**Category:** Elasticsearch\
**Created:** [January 24, 2023, 9:36am UTC](https://discuss.elastic.co/t/fscrawler-rest-with-elasticsearch/323803 "2023-01-24T09:36:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Johnson\_F](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnson_f/32/116059_2.png) [@Johnson\_F](https://discuss.elastic.co/u/Johnson_F)\
**Post date:** [January 24, 2023, 9:36am UTC](https://discuss.elastic.co/t/fscrawler-rest-with-elasticsearch/323803/1 "2023-01-24T09:36:07Z")

</div>

Dear All,

I had configured fscrawler 2.10 on ubuntu which works great, got some minor issues: -

a. why touch is required? i can understand timestamp has to be new on copied or uploaded files in directory so fscrawler looks only for newer ones? why? why it doesnt just pickup/crawl whatever new file uploaded regardless on date.

b. rest api- i have uploaded using multi-form data got below details:-

```auto
{
"ok": true,
"filename": "temp2.doc",
"url": "http://10.0.10.10:9200/es-cms/_doc/b7ba5b225841673daf7554132bca1",
"doc": {
"content": "\nTextbausteine:",
"meta": {
"date": "2021-06-07T06:49:00.000+00:00",
"modifier": "Test Autho",
"created": "2021-06-02T10:56:00.000+00:00"
},
"file": {
"extension": "doc",
"content_type": "application/msword",
"indexing_date": "2023-01-24T06:57:56.019+00:00",
"filename": "temp2.doc"
},
"path": {
"virtual": "temp2.doc",
"real": "temp2.doc"
}
}
}

```

my concern is; where is this temp2.doc file physically uploaded? cant find this file on fs: url location. so where does it really uploads?  
elasticsearch url can be used to view details form above output, but why this upload is not shown in minotoring-dicover section-kibana?

c. kibana-doscover shows only manually copied files into directory. doesnt detects or shows from rest-api why is it so?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 24, 2023, 10:26am UTC](https://discuss.elastic.co/t/fscrawler-rest-with-elasticsearch/323803/2 "2023-01-24T10:26:03Z")

</div>

Welcome!

> [@Johnson\_F](#):
>
> a. why touch is required?

That's because of the current implementation. I want to [implement a WatchService](https://github.com/dadoonet/fscrawler/issues/399) but it's not there.

> [@Johnson\_F](#):
>
> my concern is; where is this temp2.doc file physically uploaded? cant find this file on fs: url location. so where does it really uploads?

It's not uploaded as sadly Elasticsearch does not have a binary blob store like s3-like. The idea is that you share using an http server the source file somewhere.  
If you want, you can activate the `store_source` option:

[https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html#storing-binary-source-document](https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html#storing-binary-source-document)

But I don't recommend it unless you are storing very tiny documents (maximum some kilobytes).

> [@Johnson\_F](#):
>
> kibana-doscover shows only manually copied files into directory. doesnt detects or shows from rest-api why is it so?

My guess is that Kibana uses a date which is not available in case of the REST interface. Could you check if you are using a date field in the Kibana index pattern and which one is it?

---

<div class="post-metadata">

**Author:** ![Johnson\_F](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnson_f/32/116059_2.png) [@Johnson\_F](https://discuss.elastic.co/u/Johnson_F)\
**Post date:** [January 24, 2023, 10:53am UTC](https://discuss.elastic.co/t/fscrawler-rest-with-elasticsearch/323803/3 "2023-01-24T10:53:22Z")

</div>

Dear Mr. David,

Appreciate your time for the clarifications. Just need your advise further:

"It's not uploaded as sadly Elasticsearch does not have a binary blob store like s3-like. The idea is that you share using an http server the source file somewhere.  
If you want, you can activate the `store_source` But I don't recommend it unless you are storing very tiny documents (maximum some kilobytes)."  
\*\*\*\* My expectation is to store uploaded file from Rest API to file system for e.g. "/home/testweb/fscrawler-2.10/uploads" apart from getting indexed to ES and data to ES should reflect this uploaded file location in :  
"path": {  
"virtual": "temp2.doc",  
"real": "temp2.doc"

Regarding kibana i am using Time field: 'file.created' - Should I be using index\_created?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 24, 2023, 11:08am UTC](https://discuss.elastic.co/t/fscrawler-rest-with-elasticsearch/323803/4 "2023-01-24T11:08:02Z")

</div>

> [@Johnson\_F](#):
>
> My expectation is to store uploaded file from Rest API to file system for e.g. "/home/testweb/fscrawler-2.10/uploads"

That's a nice idea actually. Could you please open a feature request in the project?

> [@Johnson\_F](#):
>
> Regarding kibana i am using Time field: 'file.created' - Should I be using index\_created?

I'm wondering if the REST service should actually set the upload date in a field. Could you open a feature request for this as well?

In the meantime, you can just disable the date field in the Kibana index pattern so you don't have to pick one.

---

<div class="post-metadata">

**Author:** ![Johnson\_F](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnson_f/32/116059_2.png) [@Johnson\_F](https://discuss.elastic.co/u/Johnson_F)\
**Post date:** [January 24, 2023, 11:09am UTC](https://discuss.elastic.co/t/fscrawler-rest-with-elasticsearch/323803/5 "2023-01-24T11:09:36Z")

</div>

Thanks alot again, will do the needful. Appreciate your time again.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2023, 11:09am UTC](https://discuss.elastic.co/t/fscrawler-rest-with-elasticsearch/323803/6 "2023-02-21T11:09:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.