# Integrate ElasticSearch, Logstash and fscrawler

**URL:** <https://discuss.elastic.co/t/integrate-elasticsearch-logstash-and-fscrawler/238185>\
**Category:** Logstash\
**Created:** [June 23, 2020, 5:26am UTC](https://discuss.elastic.co/t/integrate-elasticsearch-logstash-and-fscrawler/238185 "2020-06-23T05:26:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![akshaybhuradia](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@akshaybhuradia](https://discuss.elastic.co/u/akshaybhuradia)\
**Post date:** [June 23, 2020, 5:26am UTC](https://discuss.elastic.co/t/integrate-elasticsearch-logstash-and-fscrawler/238185/1 "2020-06-23T05:26:57Z")

</div>

I have meta data for document in database.  
Example : filename, file description, file\_tags and etc

Earlier i was using this data for search. elasticsearch--for search and logstash--for accessing data source(db --\> table record)

Now i want to do also word search within document.

can i merge fscrawler with logstash

for example : add one more field content in es template and this column used by fscrawler.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 23, 2020, 7:04am UTC](https://discuss.elastic.co/t/integrate-elasticsearch-logstash-and-fscrawler/238185/2 "2020-06-23T07:04:42Z")

</div>

You can use the FSCrawler REST API and the `_simulate` endpoint from any application you could write. Then merge the result with the data coming from your database.

Not sure if this is doable with LS though.

---

<div class="post-metadata">

**Author:** ![akshaybhuradia](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@akshaybhuradia](https://discuss.elastic.co/u/akshaybhuradia)\
**Post date:** [June 23, 2020, 8:43am UTC](https://discuss.elastic.co/t/integrate-elasticsearch-logstash-and-fscrawler/238185/3 "2020-06-23T08:43:42Z")

</div>

i am using own upload server for docs

for example : (meta record for docs like doc\_id, user\_id, doc\_name, file\_description and tags) it is in one index "elsearch"

second index fscrawler(it has different fields) how do i relate this index with elsearch index.

combining one record file metadata correspond its file content.

Query : querystring like : "xyz"(this string represent file\_description or content in file)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 23, 2020, 9:00am UTC](https://discuss.elastic.co/t/integrate-elasticsearch-logstash-and-fscrawler/238185/4 "2020-06-23T09:00:04Z")

</div>

I'm saying that you can use the FSCrawler simulate endpoint to get the text from the binary. Once you have this content, you can update the existing JSon documents.

You will end up with documents looking like:

```auto
{
  "doc_id": "coming from your upload service",
  "user_id": "coming from your upload service",
  "doc_name": "coming from your upload service",
  "file_description": "coming from your upload service",
  "tags": ["coming from your upload service", "coming from your upload service"],
  "content": "This content is coming from FSCrawler simulate endpoint"
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 21, 2020, 9:00am UTC](https://discuss.elastic.co/t/integrate-elasticsearch-logstash-and-fscrawler/238185/5 "2020-07-21T09:00:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.