# Webcrawler does crawl relative url's in the same domain (incorrect protocol deny reason) **URL:** \ **Category:** Elastic Search\ **Tags:** elastic-app-search\ **Created:** [August 2, 2021, 6:32am UTC](https://discuss.elastic.co/t/webcrawler-does-crawl-relative-urls-in-the-same-domain-incorrect-protocol-deny-reason/280189 "2021-08-02T06:32:52Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![markn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markn/32/74417_2.png) [@markn](https://discuss.elastic.co/u/markn)\ **Post date:** [August 2, 2021, 6:32am UTC](https://discuss.elastic.co/t/webcrawler-does-crawl-relative-urls-in-the-same-domain-incorrect-protocol-deny-reason/280189/1 "2021-08-02T06:32:52Z")
Hi, our domain to crawl is [https://www.unive.nl](https://www.unive.nl) We run the Enterprise Search on Elastic Cloud (Azure, Netherlands) The full urls are crawled nicely. Example: [Fraudebeleid - Univé](https://www.unive.nl/overunive/fraudebeleid) Log message: Time crawler.url.deny\_reason message url.full event.type Jul 30, 2021 @ 12:13:33.000 - Indexed the document into App Search with doc\_id=6103d05f9c14a1e5fc7d36f1 [Fraudebeleid - Univé](https://www.unive.nl/overunive/fraudebeleid) info A lot of pages have a relative url's which are not crawled (in 1 crawl of 45m we got 40,460 incorrect protocol hits). The webcrawler seems to try to crawl them directly instead of adding the domain to the relative url. Example: [Verhuizing en uw verzekeringen](https://www.unive.nl/klantenservice/verhuizing) Log message: Time crawler.url.deny\_reason message url.full event.type Jul 30, 2021 @ 12:11:47.000 incorrect\_protocol - /klantenservice/verhuizing denied Btw. the webcrawler did find the full url but I think the webcrawler should also follow relative url's right?: Time crawler.url.deny\_reason message url.full event.type Jul 30, 2021 @ 11:56:50.000 - Indexed the document into App Search with doc\_id=6103ccc69c14a198d17c36c4 [Verhuizing en uw verzekeringen](https://www.unive.nl/klantenservice/verhuizing) info Is this expected behavior? I think the webcrawler should follow these relative url's also? We also added the sitemap.xml url in the search engine, but that did not help. Thanks. Kind regards, Mark ---
**Author:** ![Carlos\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carlos_d/32/126245_2.png) [@Carlos\_D](https://discuss.elastic.co/u/Carlos_D)\ **Post date:** [August 9, 2021, 11:29am UTC](https://discuss.elastic.co/t/webcrawler-does-crawl-relative-urls-in-the-same-domain-incorrect-protocol-deny-reason/280189/2 "2021-08-09T11:29:25Z")
Hi @markn: We have confirmed this is a bug in the crawler implementation. The bug has been fixed in our codebase and will be released in version 7.14.1 and 7.15 for Enterprise Search. [Stay tuned](https://github.com/elastic/elasticsearch/releases)! Thanks for reaching out, and the detailed examples that allowed us to fix this issue! ---
**Author:** ![markn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markn/32/74417_2.png) [@markn](https://discuss.elastic.co/u/markn)\ **Post date:** [August 11, 2021, 9:49am UTC](https://discuss.elastic.co/t/webcrawler-does-crawl-relative-urls-in-the-same-domain-incorrect-protocol-deny-reason/280189/3 "2021-08-11T09:49:51Z")
Thanks Carlos for confirming this and fixing it in next releases! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 8, 2021, 9:50am UTC](https://discuss.elastic.co/t/webcrawler-does-crawl-relative-urls-in-the-same-domain-incorrect-protocol-deny-reason/280189/4 "2021-09-08T09:50:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.