# Dec 20th, 2017: \[EN\]\[Elasticsearch\] Build your own Google with Elasticsearch and Gopa **URL:** \ **Category:** Advent Calendar\ **Created:** [December 19, 2017, 3:22am UTC](https://discuss.elastic.co/t/dec-20th-2017-en-elasticsearch-build-your-own-google-with-elasticsearch-and-gopa/112340 "2017-12-19T03:22:43Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![medcl.net](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl.net/32/4414_2.png) [@medcl.net](https://discuss.elastic.co/u/medcl.net)\ **Post date:** [December 19, 2017, 3:22am UTC](https://discuss.elastic.co/t/dec-20th-2017-en-elasticsearch-build-your-own-google-with-elasticsearch-and-gopa/112340/1 "2017-12-19T03:22:44Z")
Welcome to Elastic Advent Day 20, today I am going to introduce a new way to quickly build your own Google with open source [Elasticsearch](https://github.com/elastic/elasticsearch) and [Gopa](https://github.com/infinitbyte/gopa). Gopa is a open source crawler project by myself, which is written in Golang. You may have already tried other crawler software to do the same stuff, and also you may ask me “why you are reinvent the wheel?”, well, I just did it. The goal here is to build a vertical search engine, which is used to search all the elasticsearch related stuff. This is a “Google” focus on Elastic topics, and we hope to use it find Elastic articles, posts or Discuss posts in one place and so we are going to index the websites [elastic.co](https://www.elastic.co/) and [discuss.elastic.co](https://discuss.elastic.co). # Requirement - An Elasticsearch service, I am going to use [Elastic Cloud](https://cloud.elastic.co/), which is the best Elasticsearch-aaS as you can find! Of course, you can use your own Elasticsearch cluster. - [Gopa](https://github.com/infinitbyte/gopa-snapshot/releases), the crawler I am going to use in this article. - [Chrome browser](https://www.google.com/chrome/index.html), yep, we need it to work with Gopa, it is optional, but for crawling [discuss.elastic.co](http://discuss.elastic.co), we need it. # Create a cluster in Elastic Could Create a Elasticsearch cluster with Elastic Cloud is really simple! [Log into](https://cloud.elastic.co/#/authentication/login/) the console, select Elasticsearch version greater than 5.0. Woot if you are choosing 6.1! In Elastic Cloud, you can stay in the cutting edge of Elasticsearch. ![Snip20171219_7](https://us1.discourse-cdn.com/elastic/original/3X/c/2/c2166fbce179b59ca9b5e87a6b577462eaca9162.png) Save the Elasticsearch URL and password to somewhere, we will be using them later. # Get Gopa Go to [Gopa-Snapshot](https://github.com/infinitbyte/gopa-snapshot/releases) to download a pre-compiled binary. I am going to download the [`darwin64.tar.gz`](https://github.com/infinitbyte/gopa-snapshot/releases) and run it in my Macbook. Open your favourite terminal and execute the following command to download the Gopa binary: ```auto wget https://github.com/infinitbyte/gopa-snapshot/releases/download/master/darwin64.tar.gz ``` After decompress the archive, you will see some config files and the main `gopa-darwin64` binary. ```auto ➜ tar vxzf darwin64.tar x gopa-darwin64 x gopa.yml x config/ x config/elasticsearch/ x config/elasticsearch/gopa-blob-mapping.sh x config/elasticsearch/gopa-snapshot-mapping.sh x config/elasticsearch/clean-all-gopa-data.sh x config/elasticsearch/gopa-task-mapping.sh x config/elasticsearch/gopa-index-mapping.sh x config/elasticsearch/gopa-host-mapping.sh x stop.sh ``` # Config Elasticsearch We need to create a index and mapping in Elasticsearch, the script is located in Gopa’s folder that we just decompressed. Open Kibana, from the provided URL in your Elastic Cloud console, and login with your password (which we saved from the previous step). Copy index creation script from the file: `config/elasticsearch/gopa-index-mapping.sh` and execute the QueryDSL. ![Snip20171219_8](https://us1.discourse-cdn.com/elastic/original/3X/b/3/b39fcc8588dfdb47921a3262bf4046346792168c.png) This step make sure the search will work just fine. # Config Gopa Before we start the `gopa`, let’s make sure we config it correctly. Open `gopa.yml`, navigate to the following config section, and update the Elasticsearch URL and password (which we saved from Elastic Cloud in previous step). ```auto - name: index enabled: true ui: enabled: true site_name: Elasticsearch logo: https://static-www.elastic.co/cn/assets/blt6050efb80ceabd47/elastic-logo (2).svg?q=294 favicon: https://www.elastic.co/favicon.ico elasticsearch: endpoint: https://YOUR-ELASTIC-CLOUD-URL-HERE index_prefix: gopa- username: elastic password: i-am-not-gonna-tell-you ``` Next, go to the `chrome` plugin section and check if the chrome’s command path is correct for your own install. ```auto plugins: - name: chrome enabled: true command: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" debug_port: 9223 ``` Now we are all set, the next thing is to start it. # Start Gopa Start gopa is simple, execute the command below: ```auto ➜ ./gopa-darwin64 ________________ _______________ / _____/ \_____ \\ ______ \/ _ \ / \ ___/ | \|___ / /_\ \ \ \_\ \/ | \ | / | \ \ ______/\_______ / ____| \____ |__ / \/ \/ \/ [gopa] 0.10.0_SNAPSHOT ///last commit: 65e729c, Sun Dec 17 10:52:49 2017 +0800, medcl, disable chrome test for CI /// [12-19 10:53:35] [INF] [instance.go:23] workspace: data/gopa/nodes/0 [12-19 10:53:35] [INF] [pipeline.go:68] pipeline: checker started with 5 shards [12-19 10:53:35] [INF] [pipeline.go:68] pipeline: fetch started with 1 shards [12-19 10:53:35] [INF] [pipeline.go:68] pipeline: update started with 4 shards [12-19 10:53:35] [INF] [ui.go:148] http server listen at: http://127.0.0.1:9001 [12-19 10:53:35] [INF] [api.go:132] api server listen at: http://127.0.0.1:8001 ``` As you can see, the `gopa-darwin` started and listened on two ports, one is for UI, and the other one is for the API. # Rocking now Let’s open `http://127.0.0.1:9001/admin/console/` with your favorite web browser (NO IE Please). You will see a console page and we type the command to start the crawling: `seed http://elastic.co` and `seed discuss.elastic.co` ![Snip20171219_9](https://us1.discourse-cdn.com/elastic/original/3X/1/2/1226e5282979265eda1edff9d772601af54b68a3.png) That's all you need to do. Let's check out the search page, open: `http://127.0.0.1:9001/`, type `elasticsearch` and you will see the results: ![Snip20171219_12](https://us1.discourse-cdn.com/elastic/original/3X/b/e/bedaf444599f6f9e11e6bc316bd3037c5ae4f357.png) Now, a simple `Google` for your own `Elastic` is up and running. you can easily to roll it for your own purpose. # Final words Gopa is a experimental project and is not 100% stable. There are some sample configs available but not covered in this article, you may discover it by yourself. If you have any problem to use it, please file a issue [here](https://github.com/infinitbyte/gopa/issues). Like it? Give it a star! Also if you need more complete site search solution, you may also interested in our enterprise level SaaS product from our Swiftype team, [More info](https://www.elastic.co/cn/solutions/site-search) and [Here](https://app.swiftype.com/users/sign_up?utm_source=elastic). That’s it, hope you enjoyed it. ---
**Author:** ![medcl.net](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl.net/32/4414_2.png) [@medcl.net](https://discuss.elastic.co/u/medcl.net)\ **Post date:** [December 27, 2017, 12:36am UTC](https://discuss.elastic.co/t/dec-20th-2017-en-elasticsearch-build-your-own-google-with-elasticsearch-and-gopa/112340/2 "2017-12-27T00:36:11Z")
This topic was automatically closed after 7 days. New replies are no longer allowed. ---
**Author:** ![medcl.net](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl.net/32/4414_2.png) [@medcl.net](https://discuss.elastic.co/u/medcl.net)\ **Post date:** [April 10, 2018, 9:43am UTC](https://discuss.elastic.co/t/dec-20th-2017-en-elasticsearch-build-your-own-google-with-elasticsearch-and-gopa/112340/3 "2018-04-10T09:43:20Z")
---
**Author:** ![drfuka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drfuka/32/109160_2.png) [@drfuka](https://discuss.elastic.co/u/drfuka)\ **Post date:** [May 2, 2018, 8:42pm UTC](https://discuss.elastic.co/t/dec-20th-2017-en-elasticsearch-build-your-own-google-with-elasticsearch-and-gopa/112340/4 "2018-05-02T20:42:49Z")
Thanks for sharing. This is a great example! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 23, 2018, 8:31am UTC](https://discuss.elastic.co/t/dec-20th-2017-en-elasticsearch-build-your-own-google-with-elasticsearch-and-gopa/112340/5 "2018-08-23T08:31:26Z")