# Web crawler (github.com/elastic/crawler) to only fetch specific URLS

**URL:** <https://discuss.elastic.co/t/web-crawler-github-com-elastic-crawler-to-only-fetch-specific-urls/375669>\
**Category:** Elastic Search\
**Created:** [March 10, 2025, 10:17pm UTC](https://discuss.elastic.co/t/web-crawler-github-com-elastic-crawler-to-only-fetch-specific-urls/375669 "2025-03-10T22:17:36Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![Vikram\_Tiwari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikram_tiwari/32/141931_2.png) [@Vikram\_Tiwari](https://discuss.elastic.co/u/Vikram_Tiwari)\
**Post date:** [March 10, 2025, 10:43pm UTC](https://discuss.elastic.co/t/web-crawler-github-com-elastic-crawler-to-only-fetch-specific-urls/375669/2 "2025-03-10T22:43:48Z")

</div>

This fixed it: [How to index only given urls in the Elasticsearch using Open Crawler - #2 by nfeekery](https://discuss.elastic.co/t/how-to-index-only-given-urls-in-the-elasticsearch-using-open-crawler/361899/2)

You need to:

- Have `seed_urls` as the urls that you want to sync
- `sitemap_discovery_disabled: true`
- `max_crawl_depth: 1`

---

_[View the full topic](https://discuss.elastic.co/t/web-crawler-github-com-elastic-crawler-to-only-fetch-specific-urls/375669)._