# How to set up website crawl with ElasticSearch **URL:** \ **Category:** Elasticsearch\ **Created:** [January 23, 2017, 4:08pm UTC](https://discuss.elastic.co/t/how-to-set-up-website-crawl-with-elasticsearch/72507 "2017-01-23T16:08:37Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Arul\_Krishnamoorthy](https://avatars.discourse-cdn.com/v4/letter/a/7cd45c/32.png) [@Arul\_Krishnamoorthy](https://discuss.elastic.co/u/Arul_Krishnamoorthy)\ **Post date:** [January 23, 2017, 4:08pm UTC](https://discuss.elastic.co/t/how-to-set-up-website-crawl-with-elasticsearch/72507/1 "2017-01-23T16:08:37Z")
My understanding is that even with latest version of ElasticSearch (5.1.2) there is no build-in functionality for crawling a website. What are the options and any recommendations among preferred options. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 23, 2017, 4:23pm UTC](https://discuss.elastic.co/t/how-to-set-up-website-crawl-with-elasticsearch/72507/2 "2017-01-23T16:23:10Z")
Is it a public website with static pages or a private one built from database data? ---
**Author:** ![Arul\_Krishnamoorthy](https://avatars.discourse-cdn.com/v4/letter/a/7cd45c/32.png) [@Arul\_Krishnamoorthy](https://discuss.elastic.co/u/Arul_Krishnamoorthy)\ **Post date:** [January 23, 2017, 4:34pm UTC](https://discuss.elastic.co/t/how-to-set-up-website-crawl-with-elasticsearch/72507/3 "2017-01-23T16:34:46Z")
Thanks David for your reponse. Its a public website with section of the pages with dynamic data ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 23, 2017, 5:07pm UTC](https://discuss.elastic.co/t/how-to-set-up-website-crawl-with-elasticsearch/72507/4 "2017-01-23T17:07:20Z")
So I guess you don't have access to the datasource? The structured data I mean? May be using [nutch](https://nutch.apache.org/) could help? I know there are some recipes on the web about connecting Nutch and elasticsearch. ---
**Author:** ![Arul\_Krishnamoorthy](https://avatars.discourse-cdn.com/v4/letter/a/7cd45c/32.png) [@Arul\_Krishnamoorthy](https://discuss.elastic.co/u/Arul_Krishnamoorthy)\ **Post date:** [January 24, 2017, 9:19am UTC](https://discuss.elastic.co/t/how-to-set-up-website-crawl-with-elasticsearch/72507/5 "2017-01-24T09:19:39Z")
Thanks Again David. I was thinking Nutch implementation itself will be heavy weight with dependency for underlying store. Are there any lightweight options ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 24, 2017, 9:32am UTC](https://discuss.elastic.co/t/how-to-set-up-website-crawl-with-elasticsearch/72507/6 "2017-01-24T09:32:17Z")
I don't know. Never did any web crawling in the past. I'm always prefering indexing from the data source than from the rendered pages. But may be it's not possible for you. ---
**Author:** ![Arul\_Krishnamoorthy](https://avatars.discourse-cdn.com/v4/letter/a/7cd45c/32.png) [@Arul\_Krishnamoorthy](https://discuss.elastic.co/u/Arul_Krishnamoorthy)\ **Post date:** [January 24, 2017, 9:43am UTC](https://discuss.elastic.co/t/how-to-set-up-website-crawl-with-elasticsearch/72507/7 "2017-01-24T09:43:45Z")
No problem. Thank you David. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 21, 2017, 9:43am UTC](https://discuss.elastic.co/t/how-to-set-up-website-crawl-with-elasticsearch/72507/8 "2017-02-21T09:43:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.