# ElasticSearch6.4 compatible web crawler **URL:** \ **Category:** Elasticsearch\ **Created:** [October 30, 2018, 10:32am UTC](https://discuss.elastic.co/t/elasticsearch6-4-compatible-web-crawler/154619 "2018-10-30T10:32:31Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Subasini](https://avatars.discourse-cdn.com/v4/letter/s/cdc98d/32.png) [@Subasini](https://discuss.elastic.co/u/Subasini)\ **Post date:** [October 30, 2018, 10:32am UTC](https://discuss.elastic.co/t/elasticsearch6-4-compatible-web-crawler/154619/1 "2018-10-30T10:32:31Z")
Hi All, I want to use ElasticSearch 6.4 as I am going to replace GSA in my application. ElasticSearch will crawl around 60+ websites including child links. Since we are not going to use Cloud based solution - 1. Can I use nutch 2.3.1 crawler and My SQL with ElasticSearch 6.4 2. Which other softwares do I need to replace GSA with ElasticSearch? 3. Does anybody has any steps how to implement ? Please respond. It is of higher priority. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 30, 2018, 4:58pm UTC](https://discuss.elastic.co/t/elasticsearch6-4-compatible-web-crawler/154619/2 "2018-10-30T16:58:29Z")
> [@Subasini](#): > > Please respond. It is of higher priority. Read [this](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and specifically the "Also be patient" part. I personally consider that someone who has a cluster in production down is more urgent than a question about a project that does not exist yet. Anyway, some answers: 1. No idea. Never used Nutch. May be ask to the Nutch mailing list if any? 2. It depends on what are your needs. I wrote [FSCrawler](https://fscrawler.readthedocs.io/) to crawl files on disk for example and parse them with Apache Tika. 3. Too wide question. ---
**Author:** ![willjohnson](https://avatars.discourse-cdn.com/v4/letter/w/7c8e57/32.png) [@willjohnson](https://discuss.elastic.co/u/willjohnson)\ **Post date:** [October 30, 2018, 6:59pm UTC](https://discuss.elastic.co/t/elasticsearch6-4-compatible-web-crawler/154619/3 "2018-10-30T18:59:39Z")
Depending on your needs and your current GSA configuration, there really aren't OSS web crawlers out there that cover everything GSA does and what websites produce these days. Handling things like modern javascript frameworks and complex authentication are incredibly difficult to do and many commercial crawlers still struggle with those technologies. That being said, commercial offerings are probably your best bet if you really are in a hurry to solve the problem. If you did want to build your own, Nutch is a descent place to start, but be prepared to spend a _long_ making it do everything the GSA does today. As for other software, you could conceivably cobble everything together including document parsers, linguistic packages etc, but again it can take a while to make it all work together. ---
**Author:** ![Subasini](https://avatars.discourse-cdn.com/v4/letter/s/cdc98d/32.png) [@Subasini](https://discuss.elastic.co/u/Subasini)\ **Post date:** [October 31, 2018, 4:49am UTC](https://discuss.elastic.co/t/elasticsearch6-4-compatible-web-crawler/154619/4 "2018-10-31T04:49:56Z")
As per my requirement, I am trying with Nutch 2.3.1. If anybody has any documents or links regarding elastic and nutch , please share. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 28, 2018, 5:00am UTC](https://discuss.elastic.co/t/elasticsearch6-4-compatible-web-crawler/154619/5 "2018-11-28T05:00:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.