# Crawl sitemap only **URL:** \ **Category:** Elastic Search\ **Tags:** crawler, webcrawler\ **Created:** [February 27, 2025, 2:39pm UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169 "2025-02-27T14:39:52Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![pngworkforce](https://avatars.discourse-cdn.com/v4/letter/p/7c8e57/32.png) [@pngworkforce](https://discuss.elastic.co/u/pngworkforce)\ **Post date:** [February 27, 2025, 2:39pm UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169/1 "2025-02-27T14:39:52Z")
Hello! I have a question similar to the one listed here -\> [Web crawler is crawling URLs that are not on the sitemap](https://discuss.elastic.co/t/web-crawler-is-crawling-urls-that-are-not-on-the-sitemap/350533) I have added a sitemap index as the sitemap to my crawler in Elastic Cloud UI. How do I instruct the crawler to just crawl the URLs listed in the sitemap files referenced in the sitemap index? Setting connector.crawler.crawl.max\_crawl\_depth.limit to 1 doesn't work, as referenced in the post linked above, sitemaps are treated differently to entry points. Is there some other way to crawl only whats referenced in the sitemaps? Adding 1000's of links as entry points would be inefficient Thanks! Imran ---
**Author:** ![Jedr\_Blaszyk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jedr_blaszyk/32/127637_2.png) [@Jedr\_Blaszyk](https://discuss.elastic.co/u/Jedr_Blaszyk)\ **Post date:** [March 3, 2025, 10:32am UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169/2 "2025-03-03T10:32:08Z")
Hey @pngworkforce! Yes, sitemaps and entrypoints are different, so setting a sitemap as an entrypoint won’t work since the crawler treats XML just as binary files (hence it won't follow links referenced inside). However, starting a crawl with a custom sitemap (with automatic sitemap discovery disabled) should limit crawling to pages referenced in the sitemap. What do you mean by “doesn’t work”? Are you getting too many or too few documents crawled? ---
**Author:** ![pngworkforce](https://avatars.discourse-cdn.com/v4/letter/p/7c8e57/32.png) [@pngworkforce](https://discuss.elastic.co/u/pngworkforce)\ **Post date:** [March 4, 2025, 12:58am UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169/3 "2025-03-04T00:58:25Z")
Hi @Jedr_Blaszyk Thanks for the reply. We are getting too many show up in the crawl. We are occasionally even getting domains that aren't configured being added in the index (usually only one or 2 pages of other domains, but fairly common). We have a index with a single entrypoint and 2 sitemaps - one for pages and the other for docuements and other assets. All are using the same TLD. We can just exclude these extra domain results by adding that domain to the crawler and then disallowing all, but from what I understand it shouldn't crawl any documents that don't match the configured domain. Other than results from the domain we aren't expecting, there are other documents from the domain we have configured being included that aren't in the sitemap. Is there any way to crawl and strictly stick to the sitemap? Cheers ---
**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\ **Post date:** [March 4, 2025, 9:43am UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169/4 "2025-03-04T09:43:48Z")
Hi @pngworkforce If your site is public, can you share the domain so we can analyse it more clearly? Also, can you provide the following information? 1. What version of Elasticsearch/Enterprise Search are you using? 2. Are you using App Search Crawler or Elastic Crawler? 3. Do you schedule full crawls, or run them manually through the UI? 4. Do you have any scheduled partial crawls? (these are set through the "Crawl with custom settings" UI button) 5. What other Crawler server configurations have you set aside from max crawl depth? 6. Are the sitemaps located on the same domain as the "main" domain for this crawler? 7. You mentioned some domains are being mistakenly ingested. Are these subdomains, or completely different domains? (e.g. if the main domain is `xyz.com`, is the other domain a subdomain like `abc.xyz.com` or completely different like `123.com`) ---
**Author:** ![pngworkforce](https://avatars.discourse-cdn.com/v4/letter/p/7c8e57/32.png) [@pngworkforce](https://discuss.elastic.co/u/pngworkforce)\ **Post date:** [March 4, 2025, 10:54pm UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169/5 "2025-03-04T22:54:11Z")
Hi @nfeekery The site isn't publicly available unfortunately. Answers to your questions below > [@nfeekery](#): > > - What version of Elasticsearch/Enterprise Search are you using? We are using 8.16.0 > [@nfeekery](#): > > - Are you using App Search Crawler or Elastic Crawler? Elastic web crawler in the cloud UI > [@nfeekery](#): > > - Do you schedule full crawls, or run them manually through the UI? We do both. We have a full crawl on a 3 hr update schedule and we often manually crawl. > [@nfeekery](#): > > - Do you have any scheduled partial crawls? (these are set through the "Crawl with custom settings" UI button) No, we don't schedule partial crawls > [@nfeekery](#): > > - What other Crawler server configurations have you set aside from max crawl depth? We have also tried setting the link extraction to 0 (which is out of range) and 1 ```auto connector.crawler.crawl.max_crawl_depth.limit: 1 connector.crawler.extraction.extracted_links_count.limit: 1 ``` > [@nfeekery](#): > > - Are the sitemaps located on the same domain as the "main" domain for this crawler? Yes, sitemaps are in the root of the domain web folder > [@nfeekery](#): > > - You mentioned some domains are being mistakenly ingested. Are these subdomains, or completely different domains? (e.g. if the main domain is `xyz.com`, is the other domain a subdomain like `abc.xyz.com` or completely different like `123.com`) Yes, the domain being included is another subdomain For example, our entry point domain is `test-www.website.com` Our sitemaps are `test-www.website.com/sitemap.xml` and `test-www.website.com/sitemap-assets.xml`. Both of these are sitemap index files containing links to other sitemaps for each site section The domain being found in our results alongside the expected results is `test-assets.website.com`. There aren't many of these in the index (about 50 out of 10,000 docs) but they do show up in query hits regularly. They also have the same URL path as results from the correct `test-www.website.com` domain. We have checked the sitemaps and canonical URLs, this domain does not exist anywhere in either. ---
**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\ **Post date:** [March 10, 2025, 10:47am UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169/6 "2025-03-10T10:47:27Z")
@pngworkforce Thanks for providing all of that information. It's certainly unexpected behaviour. The server settings you've set should work as expected. I tried reproducing a similar setup but I couldn't reproduce it, so there must be something that we're missing here. I have some more follow up questions. 1. The incorrect `test-assets.website.com` URLs must be being discovered by something. Do the URL links on pages under `test-www.website.com` have links to `test-assets.website.com`? 2. Has this index ever previously been used to crawl URLs under the domain `test-assets.website.com`? 3. What do you have registered in your "Domains" section? 4. What crawl rules have you configured? 5. If you create a new web crawler for a new index and crawl the same website, do you still ingest the same incorrect data? ---
**Author:** ![pngworkforce](https://avatars.discourse-cdn.com/v4/letter/p/7c8e57/32.png) [@pngworkforce](https://discuss.elastic.co/u/pngworkforce)\ **Post date:** [March 16, 2025, 9:46am UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169/7 "2025-03-16T09:46:09Z")
Hi @nfeekery , answers to your questions below > 1. The incorrect `test-assets.website.com` URLs must be being discovered by something. Do the URL links on pages under `test-www.website.com` have links to `test-assets.website.com`? Yes, some pages do have this linked from them > 1. Has this index ever previously been used to crawl URLs under the domain `test-assets.website.com`? No, never set up for this site > 1. What do you have registered in your "Domains" section? Just `https://test-www.website.com` > 1. What crawl rules have you configured? No specific crawl rules. Just the allow all default > 1. If you create a new web crawler for a new index and crawl the same website, do you still ingest the same incorrect data? Yes, we were albe to reproduce this issue in a separate index with the same settings. ---
**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\ **Post date:** [March 20, 2025, 7:53am UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169/8 "2025-03-20T07:53:55Z")
@pngworkforce Thanks for providing the extra information. It sounds like you might have found a bug. By all accounts the way you're describing your setup, it should work. If I were to guess at the cause, I think Crawler isn't properly rejecting URLs from other subdomains, possibly because they share a main domain. In the meantime, you mentioned that there is a workaround, which is to add `test-assets.website.com` as a domain and give it a crawl rule rejecting all URLs. Could you continue doing this while we investigate on our end? Sorry for the inconvenience and thank you again for providing all of the extra information I requested. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 17, 2025, 7:54am UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169/9 "2025-04-17T07:54:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.