# Crawling web sites and indexing the extracted content

**URL:** <https://discuss.elastic.co/t/crawling-web-sites-and-indexing-the-extracted-content/3545>\
**Category:** Elasticsearch\
**Created:** [November 10, 2010, 12:23pm UTC](https://discuss.elastic.co/t/crawling-web-sites-and-indexing-the-extracted-content/3545 "2010-11-10T12:23:25Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [November 11, 2010, 3:28pm UTC](https://discuss.elastic.co/t/crawling-web-sites-and-indexing-the-extracted-content/3545/4 "2010-11-11T15:28:01Z")

</div>

Hi Otis,  
yea, I noticed that in your Lucene Revolution presentation. My question  
would be how much do you manipulate the HTML document content? Do you  
extract all data from it or do you extract just some specific portions?  
Which tools do you use for this?  
Thanks,  
Lukas

On Thu, Nov 11, 2010 at 1:49 PM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote:

> Hi Lukáš,
> 
> We use Droids for [http://search-hadoop.com](http://search-hadoop.com) and [http://search-lucene.com](http://search-lucene.com)  
> Yes, the community is slim and not very active, but what's there seems  
> to work. It's _much_ simpler than Nutch or Heritrix.
> 
> ## Otis
> 
> Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch  
> Lucene ecosystem search :: [http://search-lucene.com/](http://search-lucene.com/)
> 
> On Nov 10, 7:23 am, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote:
> 
> > Hi,
> > 
> > does anybody have any experience crawling web content and indexing it  
> > with  
> > Elastic Search?
> > 
> > I need to crawl relatively small number of web pages from few internet  
> > domain and extract some content from these web pages. Rather then massive  
> > scale of the crawler I need very precise control and possibility to  
> > implement custom code.
> > 
> > I need very flexible control of the content extraction and processing, at  
> > least the following:
> > 
> > 1. extracting just a portion of crawled html page (for example only DIV  
> > elements with specific class or id attributes)
> > 2. extracting only specific links from the content identified in #1
> > 3. add metadata for each extracted content
> > 4. either storing the output into file system or possibility to process  
> > it  
> > and send to search server (e.g. indexing the extracted content with  
> > search  
> > server)
> > 
> > I think that with my requirements I will end up writing some (possibly  
> > not a  
> > lot of) custom code. That is ok but would like to avoid unnecessary  
> > complexity if possible. I do not consider using incremental crawling (it  
> > is  
> > fine to reindex every time).
> > 
> > There are may web crawlers out there... but they are either old with  
> > inactive community or they tend to become complex (sure, web crawling is  
> > a  
> > complex thing...).  
> > There are probably two good candidates: Heritrix and Nutch. Would you  
> > recommend using one of them for the above requirements?  
> > There is also Apache Droids project and it should allow for custom  
> > crawlers  
> > but it seems to me that the project is quite inactive.
> > 
> > Any comments are welcome.
> > 
> > Regards,  
> > Lukas

---

_[View the full topic](https://discuss.elastic.co/t/crawling-web-sites-and-indexing-the-extracted-content/3545)._