# Webcrawl one level

**URL:** <https://discuss.elastic.co/t/webcrawl-one-level/356000>\
**Category:** Elastic Search\
**Tags:** elastic-app-search\
**Created:** [March 22, 2024, 3:30pm UTC](https://discuss.elastic.co/t/webcrawl-one-level/356000 "2024-03-22T15:30:55Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![etp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/etp/32/117415_2.png) [@etp](https://discuss.elastic.co/u/etp)\
**Post date:** [March 22, 2024, 3:30pm UTC](https://discuss.elastic.co/t/webcrawl-one-level/356000/1 "2024-03-22T15:30:55Z")

</div>

Hi,  
I was wondering if one can configure a web crawler such that only the immediate links on the page corresponding to the entry point will be crawled and the rest should be skipped.  
Eg.  
Entry\_\_page has Link1 and Link2 -\> both sub-links should be crawled  
Link1\_page has Link3 -\> should not be considered

i have considered crawl rules, but there is no generic/simple text in the url that will allow to differentiate un-needed links.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\
**Post date:** [March 22, 2024, 3:38pm UTC](https://discuss.elastic.co/t/webcrawl-one-level/356000/2 "2024-03-22T15:38:08Z")

</div>

Hi @etp

You can configure this two ways. If you want this to be a set change for all crawlers you can update the [Enterprise Search configuration](https://www.elastic.co/guide/en/enterprise-search/8.12/configuration.html#configuration-settings-elastic-crawler-per-crawl-resource-limits), just change the value for `connector.crawler.crawl.max_crawl_depth.limit`.

You can also do this per-crawl as well if you choose to [crawl with custom settings](https://www.elastic.co/guide/en/enterprise-search/current/crawler-managing.html#crawler-crawl-custom-settings).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2024, 3:38pm UTC](https://discuss.elastic.co/t/webcrawl-one-level/356000/3 "2024-04-19T15:38:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.