# How to index only given urls in the Elasticsearch using Open Crawler

**URL:** <https://discuss.elastic.co/t/how-to-index-only-given-urls-in-the-elasticsearch-using-open-crawler/361899>\
**Category:** Elastic Search\
**Tags:** crawler\
**Created:** [June 23, 2024, 10:44am UTC](https://discuss.elastic.co/t/how-to-index-only-given-urls-in-the-elasticsearch-using-open-crawler/361899 "2024-06-23T10:44:42Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\
**Post date:** [June 24, 2024, 8:04am UTC](https://discuss.elastic.co/t/how-to-index-only-given-urls-in-the-elasticsearch-using-open-crawler/361899/2 "2024-06-24T08:04:34Z")

</div>

Hi @jahedi

> I did try the option `max_crawl_depth: 1` on the config file but it didn't work properly.

In theory the `max_crawl_depth: 1` should be enough to do this. Can you describe what happened? How many docs did it end up ingesting?

One possibility is the Crawler can also use the `sitemap.xml` for content discovery. Do you have this file in your website? If so, you can set `sitemap_discovery_disabled: true` in the config to ignore it.

If that isn't the case, this is likely a bug.

---

_[View the full topic](https://discuss.elastic.co/t/how-to-index-only-given-urls-in-the-elasticsearch-using-open-crawler/361899)._