# Elasticsearch Crawling

**URL:** <https://discuss.elastic.co/t/elasticsearch-crawling/107257>\
**Category:** Elastic Community and Ecosystem\
**Created:** [November 11, 2017, 12:02am UTC](https://discuss.elastic.co/t/elasticsearch-crawling/107257 "2017-11-11T00:02:46Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![GeorgeP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgep/32/24095_2.png) [@GeorgeP](https://discuss.elastic.co/u/GeorgeP)\
**Post date:** [November 11, 2017, 12:02am UTC](https://discuss.elastic.co/t/elasticsearch-crawling/107257/1 "2017-11-11T00:02:47Z")

</div>

Hello guys,

I was checking "Google" but I didn't find too much info about it, so any help would be greatly appreciated.  
I was working with Google Search Appliance before but it was decommissioned, and Elastic seems to be the best option on the market right now.

I need to:

- Crawl & index a number of websites (around 50)
- Serve them as an XML to a web application

The current version of Elasticsearch does not have a crawler? I need to install something else?

I am using the Elastic Cloud in AWS.

Thank you!

---

<div class="post-metadata">

**Author:** ![GeorgeP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgep/32/24095_2.png) [@GeorgeP](https://discuss.elastic.co/u/GeorgeP)\
**Post date:** [November 11, 2017, 1:20am UTC](https://discuss.elastic.co/t/elasticsearch-crawling/107257/2 "2017-11-11T01:20:58Z")

</div>

\*We will crawl only public websites, like Twitter accounts etc.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 11, 2017, 2:40am UTC](https://discuss.elastic.co/t/elasticsearch-crawling/107257/3 "2017-11-11T02:40:33Z")

</div>

[https://swiftype.com/](https://swiftype.com/) can do most of that!

---

<div class="post-metadata">

**Author:** ![GeorgeP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgep/32/24095_2.png) [@GeorgeP](https://discuss.elastic.co/u/GeorgeP)\
**Post date:** [November 12, 2017, 10:58pm UTC](https://discuss.elastic.co/t/elasticsearch-crawling/107257/4 "2017-11-12T22:58:02Z")

</div>

> [@GeorgeP](#):
>
> I need to:
> 
> Crawl & index a number of websites (around 50)  
> Serve them as an XML to a web application

Thanks Mark!  
Swiftype is ok, but unfortunately we need to have more or less a real-time crawling (every 5 minutes) and their solution (the cheap one with \<100$/month) offers only 1 crawling every 3 days or so.

I was checking now some other solutions like: [80legs.com](http://80legs.com) and if I find something I will post here.

Can ElsticSearch be used for this type of live crawling, indexing & serving?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 12, 2017, 11:32pm UTC](https://discuss.elastic.co/t/elasticsearch-crawling/107257/5 "2017-11-12T23:32:50Z")

</div>

Elasticsearch can be the backend for storing the data collected from crawlers, but it has no crawling capabilities.

---

<div class="post-metadata">

**Author:** ![lukas\_vlcek](https://avatars.discourse-cdn.com/v4/letter/l/dfb087/32.png) [@lukas\_vlcek](https://discuss.elastic.co/u/lukas_vlcek)\
**Post date:** [November 13, 2017, 8:23am UTC](https://discuss.elastic.co/t/elasticsearch-crawling/107257/6 "2017-11-13T08:23:17Z")

</div>

May be you can have a look at [https://github.com/DigitalPebble/storm-crawler](https://github.com/DigitalPebble/storm-crawler) it has some integration with ES. (I haven't used it myself though.)

Regards,  
Lukáš

---

<div class="post-metadata">

**Author:** ![GeorgeP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgep/32/24095_2.png) [@GeorgeP](https://discuss.elastic.co/u/GeorgeP)\
**Post date:** [November 13, 2017, 11:47pm UTC](https://discuss.elastic.co/t/elasticsearch-crawling/107257/7 "2017-11-13T23:47:12Z")

</div>

@warkolm Yes.. I was a bit shocked to see that there is no official crawler (at least on the Cloud version).  
It's like selling only the engine and some other parts of a car 😄, but you need to find the wheels by your own.

I guess I was too accustomed with the Google Search Appliance.

I will keep researching and post here whatever solution I find!

@lukas_vlcek Thanks! I will test it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 11, 2017, 11:47pm UTC](https://discuss.elastic.co/t/elasticsearch-crawling/107257/8 "2017-12-11T23:47:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.