To store and search a File content using elastic search

[Romy_Bobby]

Hi,
My requirement is an FAQ in my application where I require to Upload FAQ documents and later search and find these documents using the keywords in search cloumn. I Would like to know how to store and search a File content in Elastic search. Currently I am Using Firebase and Node.js for my application. I have created an account and your Hosted Elastic search . I couldn't find any documentation or proper instruction in your page for the implementation of File upload search. KIndly provide a solution

[dadoonet (David Pilato)]

There's no file upload in elasticsearch.

You can use the ingest attachment plugin.

There an example here: https://www.elastic.co/guide/en/elasticsearch/plugins/current/using-ingest-attachment.html

PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data"
      }
    }
  ]
}
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
GET my_index/_doc/my_id

The data field is basically the BASE64 representation of your binary file.

You can use FSCrawler. There's a tutorial to help you getting started. FSCrawler also exposes a REST Service which can be used for file upload. See https://fscrawler.readthedocs.io/en/latest/user/rest.html

[Romy_Bobby]

HI,
In the above explanation the file can be retrieved by its id or can we retrieve the file by its content

[dadoonet (David Pilato)]

Both.

[defalt]

Elasticsearch has a huge amount of search queries:
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html
Hope you find what you are looking for.

[Romy_Bobby]

hi,
Is it Possible to use LogStash for the searching file stored in firebase . if yes ,please guide me in the pipeline structure and share documentation link as i am little confused with it.

[defalt]

Logstash is not used for searching. It is used to insert data into elasticsearch. You use Elasticsearch for searching. You can send search requests through Kibana, Postman (windows) or CURL.

[Romy_Bobby]

so by using logstash i can insert files from firebase and later use search from elastic search>??

[defalt]

Yes

[Romy_Bobby]

LogStash uses pipeline structure right. can you please provide the implementation guide of logstash and the valid pipeline structure

[defalt]

https://www.elastic.co/guide/en/logstash/current/pipeline.html
Here is the logstash documentation: https://www.elastic.co/guide/en/logstash/current/index.html
What a pipeline can look like:

input {
	elasticsearch {
		hosts => "localhost"
		index => "Index"
	}
}

filter {
   date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS"]
	target => "@timestamp"
  }
}

output {
stdout { codec => rubydebug }
	elasticsearch {
		hosts => ["172.22.23.111"]
		document_type => "intexName"
		index => "IndexName"
	}
}

Good luck

[Romy_Bobby]

thank you