# How do you tell ES Web Crawler to stop crawling a parent's child webpages that don't include parent's nameURL name

**URL:** <https://discuss.elastic.co/t/how-do-you-tell-es-web-crawler-to-stop-crawling-a-parents-child-webpages-that-dont-include-parents-nameurl-name/372074>\
**Category:** Elastic Search\
**Created:** [December 17, 2024, 1:57pm UTC](https://discuss.elastic.co/t/how-do-you-tell-es-web-crawler-to-stop-crawling-a-parents-child-webpages-that-dont-include-parents-nameurl-name/372074 "2024-12-17T13:57:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![langelel](https://avatars.discourse-cdn.com/v4/letter/l/47e85d/32.png) [@langelel](https://discuss.elastic.co/u/langelel)\
**Post date:** [December 17, 2024, 1:57pm UTC](https://discuss.elastic.co/t/how-do-you-tell-es-web-crawler-to-stop-crawling-a-parents-child-webpages-that-dont-include-parents-nameurl-name/372074/1 "2024-12-17T13:57:37Z")

</div>

Problem:  
We are running into a problem using the ES web crawler -\> where we set the rules when it hits a specific webpage e.g., [disallow] [contains] [healthtopics] – don’t crawl that page and any of its “child” pages.

Observation:  
Currently, we are experiencing:  
When ES starts crawling the domain/seed URL -\> https://domain\_name.com/  
It accesses the url https://domain\_name.com/healthtopics.html -\> it does NOT crawl this web page “healthtopics.html”.  
But this https://domain\_name.com/healthtopics.html web page contains children web pages that don’t have “healthtopics” in the URL – so the web crawler crawls all the children web pages that don't have “healthtopics” in the url.  
E.g., here are some examples of children urls under this parent https://domain\_name.com/healthtopics.html url.  
https://domain\_name.com/bloodheartandcirculation.html  
https://domain\_name.com/brainandnerves.html

ASK:  
Do you know away through the ES web crawler -\> “Crawl rules” tab that will tell the ES web crawler -\> once you hit a URL that’s in the DO NOT crawl rules list – STOP crawling -\> go back to the proceeding URL?  
E.g., when the web crawler hits this URL -\> https://domain\_name.com/healthtopics.html STOP!! Go Back.

---

<div class="post-metadata">

**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\
**Post date:** [December 24, 2024, 9:46am UTC](https://discuss.elastic.co/t/how-do-you-tell-es-web-crawler-to-stop-crawling-a-parents-child-webpages-that-dont-include-parents-nameurl-name/372074/2 "2024-12-24T09:46:41Z")

</div>

Hi @langelel,

If you have set up a crawl rule for `[disallow] [contains] [healthtopics]`, then Crawler won't attempt to crawl any URL links on the `healthtopics` page because it won't ingest the page.

If these child pages of `healthtopics` are still being ingested, then they are still "discoverable" by Crawler. There are a few ways this could happen:

1. Other pages in the site link to those pages
2. They are included in the sitemap.xml

So my recommendation is to either try and see how Crawler is finding these pages, or to write more crawl rules that cover these pages.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 21, 2025, 9:46am UTC](https://discuss.elastic.co/t/how-do-you-tell-es-web-crawler-to-stop-crawling-a-parents-child-webpages-that-dont-include-parents-nameurl-name/372074/3 "2025-01-21T09:46:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.