# Which web crawler works best with ES

**URL:** <https://discuss.elastic.co/t/which-web-crawler-works-best-with-es/53085>\
**Category:** Elasticsearch\
**Created:** [June 17, 2016, 3:54am UTC](https://discuss.elastic.co/t/which-web-crawler-works-best-with-es/53085 "2016-06-17T03:54:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yongyao\_Jiang](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yongyao\_Jiang](https://discuss.elastic.co/u/Yongyao_Jiang)\
**Post date:** [June 17, 2016, 3:54am UTC](https://discuss.elastic.co/t/which-web-crawler-works-best-with-es/53085/1 "2016-06-17T03:54:29Z")

</div>

Hey everyone,

I am evaluating which crawler to use with ES. Do you guys have any experience or suggestions? Just did some research, and I found choices include Nutch and River Web. Personally, I don't want to involve another software such as HBase.

Thanks,  
Cody

---

<div class="post-metadata">

**Author:** ![Brucey](https://avatars.discourse-cdn.com/v4/letter/b/ccd318/32.png) [@Brucey](https://discuss.elastic.co/u/Brucey)\
**Post date:** [April 29, 2017, 3:13am UTC](https://discuss.elastic.co/t/which-web-crawler-works-best-with-es/53085/2 "2017-04-29T03:13:22Z")

</div>

Did you ever get a crawler working with ES?

I hear Nutch works with ES 2.X but I'm not really interested in going backwards.

---

<div class="post-metadata">

**Author:** ![Yongyao\_Jiang](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yongyao\_Jiang](https://discuss.elastic.co/u/Yongyao_Jiang)\
**Post date:** [April 29, 2017, 3:39am UTC](https://discuss.elastic.co/t/which-web-crawler-works-best-with-es/53085/3 "2017-04-29T03:39:42Z")

</div>

I ended up using nutch. And yes , nutch only works with 2.3 at this point. Since 2.3 has all functions I need, I'm fine with it. I recently found scrapy very powerful. May be worth a try and writing some indexing code on your own though.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:00pm UTC](https://discuss.elastic.co/t/which-web-crawler-works-best-with-es/53085/4 "2017-07-05T22:00:48Z")

</div>