How to index my eCommerce website

[saurabhbhatt]

My Requirement: I am using .NET C# for coding preference using Visual Studio 2017 and working to integrate Elasticsearch with my eCommerce website so that my website users can easily search my website.

My Current Understanding: I am new to elastic search and have found that for using Elastic search as database to search, first I need to add data to it (elasticsearch). I saw 2 methods to post data

1. using POST verb for adding data JSON data manually
2. using some data dumps like for example XML dump file as data source which would be converted and saved as JSON data in ElasticSearch. (Reference: https://www.red-gate.com/simple-talk/dotnet/net-development/how-to-build-a-search-page-with-elasticsearch-and-net/). For achieving this I will need to create .NET application that would extract the XML files and convert them to JSON and then POST.

My Question: I am confused that is it necessary that for my eCommerce site to be indexed/inserted to Elastic search, I have to create some specific custom JSON object manually which I need to insert using POST manually ? Or is there some "automatic" way wherein (similar to google) I can simply give URL of my website and Elasticsearch would crawl all the links and index/insert data automatically and this data I can then use for search results ?

Please help me in my understanding of implementation of Elasticsearch. Also if there are other popular ways to insert data into Elasticsearch than my guide me for the same. Your help is appreciated.

[dadoonet (David Pilato)]

I am confused that is it necessary that for my eCommerce site to be indexed/inserted to Elastic search

Yes. Elasticsearch only searches within its indices. Which means that something needs to send your data to it.

Or is there some "automatic" way wherein (similar to google) I can simply give URL of my website and Elasticsearch would crawl all the links and index/insert data automatically and this data I can then use for search results ?

This is a crawler indeed. The problem with crawlers IMO is that it's hard to extract structured data out of it. It's better IMO to extract the data from the source (ie the database) instead of from the rendered view (ie the html page).

Anyway, if you are looking for a Google Site Search replacement, I'd encourage you looking at https://swiftype.com/

[saurabhbhatt]

Thank you @dadoonet for your response!

dadoonet: The problem with crawlers IMO is that it's hard to extract structured data out of it. It's better IMO to extract the data from the source (ie the database) instead of from the rendered view (ie the html page).

Saurabh: Ok, I understand that I need to add data in a structured way so that my search control is better. However the data in my DB is SQL relational data with master, parent, child type breakup.

My question is, would this type of data source work fine for "ElasticSearch" ? I ask this because I think based on my read this used JSON database of NoSql type. So if simply try to import data from SQL database to ElasticSearch database, would the data be structured automatically ? Or should I create some windows service or some application that would extract data from my SQL database, create JSON structure and then submit this to "ElasticSearch"? Also currently I am already creating XML files using windows services from SQL DB. Can these XML files also serve as data source for "ElasticSearch" ? I did read somewhere that yes I can use XML files, so if anyone could please guide me to the process with reference links, that would be helpful.

[dadoonet (David Pilato)]

So if simply try to import data from SQL database to Elasticsearch database, would the data be structured automatically ?

No.

Or should I create some windows service or some application that would extract data from my SQL database, create JSON structure and then submit this to "Elasticsearch"?

Yes.

Also currently I am already creating XML files using windows services from SQL DB. Can these XML files also serve as data source for "Elasticsearch" ?

Can be. But you need to convert them to JSON anyway.

BTW, here is a blog post I wrote about moving from a SQL based search to elasticsearch: https://david.pilato.fr/blog/2015-05-09-advanced-search-for-your-legacy-application/