# What can be used to fetch files from one location and store it in elasticsearch

**URL:** <https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379>\
**Category:** Elasticsearch\
**Created:** [February 28, 2020, 7:54am UTC](https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379 "2020-02-28T07:54:30Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\
**Post date:** [February 28, 2020, 7:54am UTC](https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379/1 "2020-02-28T07:54:31Z")

</div>

I would like to know which techinque can be used to fetch the files from certain location and store it in elastic seach

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 28, 2020, 2:54pm UTC](https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379/2 "2020-02-28T14:54:54Z")

</div>

What kind of files?

- Logs? -\> Look at filebeat
- PDF, MSOffice, ... ? -\> Look at FSCrawler

---

<div class="post-metadata">

**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\
**Post date:** [March 2, 2020, 4:09am UTC](https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379/4 "2020-03-02T04:09:11Z")

</div>

Thanks for the reply.  
Is there any documentation for FSCrawler?  
Can we use FSCrawler to fetch files from ftp location and store in elastic search?  
I need some example for that. can u please help me with that

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 2, 2020, 6:38am UTC](https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379/5 "2020-03-02T06:38:37Z")

</div>

> [@samanvi\_sweety](#):
>
> Is there any documentation for FSCrawler?

[https://fscrawler.readthedocs.io/en/latest/](https://fscrawler.readthedocs.io/en/latest/)

> [@samanvi\_sweety](#):
>
> Can we use FSCrawler to fetch files from ftp location and store in Elasticsearch?

Not directly yet.

> <https://github.com/dadoonet/fscrawler/issues/147>
>
> It would be nice if we are able to lookup files over ftp. Is this something that… is already existing or something that should be as an enhancement?
> 
> \`\`\`
> "server" : {
> "hostname" : "ftp.abc.xyz",
> "port" : 21,
> "username":abc,
> "password":xyz,
> "protocol" : "ftp"
> }
> \`\`\`

You can mount hopefully your dir instead and use the "local FS" instead.

---

<div class="post-metadata">

**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\
**Post date:** [March 2, 2020, 8:37am UTC](https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379/7 "2020-03-02T08:37:52Z")

</div>

Thanks.Right now I need to fetch files from ftp location only.  
So is there any other way to fecth files from ftp and store in Elastic search?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 2, 2020, 9:25am UTC](https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379/8 "2020-03-02T09:25:20Z")

</div>

Not anything I'm aware of.

---

<div class="post-metadata">

**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\
**Post date:** [March 2, 2020, 9:57am UTC](https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379/9 "2020-03-02T09:57:25Z")

</div>

Okay thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 30, 2020, 9:57am UTC](https://discuss.elastic.co/t/what-can-be-used-to-fetch-files-from-one-location-and-store-it-in-elasticsearch/221379/10 "2020-03-30T09:57:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.