# Sitemap for example.com located at sub.example.com **URL:** \ **Category:** Elastic Search\ **Tags:** elastic-site-search\ **Created:** [January 29, 2021, 11:04am UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631 "2021-01-29T11:04:05Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Marcelo\_da\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelo_da_silva/32/77245_2.png) [@Marcelo\_da\_Silva](https://discuss.elastic.co/u/Marcelo_da_Silva)\ **Post date:** [January 29, 2021, 11:04am UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/1 "2021-01-29T11:04:05Z")
Hi, the sitemap of a website is located on a subdomain. Something like - [example.com](http://example.com) (main) - [sub.example.com/sitemap\_index.xml](http://sub.example.com/sitemap_index.xml) (sitemap for [example.com](http://example.com)) - [example.com/robots.txt](http://example.com/robots.txt) (it has a Sitemap key indicating the location of the sitemap) But it seems that the Swiftype crawler does not recognize the sitemap on robots.txt. And, from the dashboard, I can't setup the sitemap on another domain (the domain part is read only). Do you know how I can configure it to read properly from the sitemap? ---
**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\ **Post date:** [January 29, 2021, 10:17pm UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/2 "2021-01-29T22:17:30Z")
Hi @Marcelo_da_Silva ! have you configured `sub.example.com` as one of your domains? Sub-domains are not automatically included, each discrete domain must be added individually. ---
**Author:** ![Marcelo\_da\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelo_da_silva/32/77245_2.png) [@Marcelo\_da\_Silva](https://discuss.elastic.co/u/Marcelo_da_Silva)\ **Post date:** [February 3, 2021, 4:15pm UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/3 "2021-02-03T16:15:14Z")
Thanks for your reply @Sean_Story I'm on the standard plan, so, I can use only a single domain. If I add my "[sub.example.com](http://sub.example.com)" domain only, do the crawler gets the pages from "[example.com](http://example.com)" inside the sitemap? Because, all the links for "[example.com](http://example.com)" are on "[sub.example.com/sitemap\_index.xml](http://sub.example.com/sitemap_index.xml)". To add a better context, the "sub" domain is were my CMS lives, so, all pages are created on that side and that's why it's easier to leave the sitemap there. ---
**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\ **Post date:** [February 3, 2021, 7:58pm UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/4 "2021-02-03T19:58:11Z")
@Marcelo_da_Silva sounds like you need to upgrade your plan level, if you want to crawl two domains. You could do `sub.example.com` only, but you would _not_ get the pages from just `example.com`. The other option would be to put your sitemap for `example.com` on `example.com` (but you still wouldn't get pages from `sub.example.com`). ---
**Author:** ![Marcelo\_da\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelo_da_silva/32/77245_2.png) [@Marcelo\_da\_Silva](https://discuss.elastic.co/u/Marcelo_da_Silva)\ **Post date:** [February 4, 2021, 7:54am UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/5 "2021-02-04T07:54:00Z")
Hummm, "[sub.example.com](http://sub.example.com)" does not have any pages to be crawled, all pages in the sitemap are from "[example.com](http://example.com)". In that case, I have to upgrade and add 2 domains just to have make crawler finds my sitemap: - add "[example.com](http://example.com)" so the pages can be recognized and the documents created - add "[sub.example.com](http://sub.example.com)" so the sitemap can be recognized (and only that, since will be no documents on that domain) Is that correct? ---
**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\ **Post date:** [February 4, 2021, 4:22pm UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/6 "2021-02-04T16:22:45Z")
> [@Marcelo\_da\_Silva](#): > > I have to upgrade and add 2 domains just to have make crawler finds my sitemap That's definitely one option. The other option is to move or copy your sitemap to `example.com`, if there's no content on `sub.example.com` that you want crawled (other than the current sitemap). ---
**Author:** ![Marcelo\_da\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelo_da_silva/32/77245_2.png) [@Marcelo\_da\_Silva](https://discuss.elastic.co/u/Marcelo_da_Silva)\ **Post date:** [February 12, 2021, 8:14am UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/7 "2021-02-12T08:14:58Z")
The weird part is that, when I add the [example.com](http://example.com) Swiftype recognizes the sitemap url on the subdomain, even displaying the sitemap url on the success screen. But then when the crawler runs, it seems to not use that sitemap. ---
**Author:** ![Brian\_McGue](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brian_mcgue/32/63366_2.png) [@Brian\_McGue](https://discuss.elastic.co/u/Brian_McGue)\ **Post date:** [February 17, 2021, 7:48pm UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/8 "2021-02-17T19:48:42Z")
Hi @Marcelo_da_Silva, Ya, that's definitely confusing. What's likely happening is it's pulling the sitemap URL from the robots.txt file associated with your site. I can't confirm because I don't know what your robots.txt looks like. The problem then comes when the crawler goes to fetch the sitemap and it's not on the same subdomain. So we see it, but we can't fetch it. Unfortunately, your options remain as Sean described. If hosting your sitemap on the `example.com` domain is not convenient for you, you will have to upgrade in order to crawl it. Best, Brian ---
**Author:** ![Marcelo\_da\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelo_da_silva/32/77245_2.png) [@Marcelo\_da\_Silva](https://discuss.elastic.co/u/Marcelo_da_Silva)\ **Post date:** [February 22, 2021, 10:07am UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/9 "2021-02-22T10:07:44Z")
Hello Brian. That's exactly what's happening. My sitemap url is on my robots.txt and it seems that the Swiftype dashboard recognizes it during the setup. But when it is running the crawler, the sitemap is ignored completely as it lives on another subdomain. This seems to be a bug from your side as the behavior is very confusing for the users. At the moment, I created a proxy to be able to access the sitemap over the '[example.com](http://example.com)' without need to move the entire sitemap to it, so, it should be okay! ---
**Author:** ![Marcelo\_da\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelo_da_silva/32/77245_2.png) [@Marcelo\_da\_Silva](https://discuss.elastic.co/u/Marcelo_da_Silva)\ **Post date:** [February 22, 2021, 10:55am UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/10 "2021-02-22T10:55:34Z")
Another thing I noticed was that: - my sitemap was named like: '[example.com/sitemap\_index.xml](http://example.com/sitemap_index.xml)' - during setup, it was recognized properly - during crawler it seems it wasn't been used, then I went o check the config - there was a "custom sitemap" URL there with "[example.com/sitemap.xml](http://example.com/sitemap.xml)" on it So, it seems that, the setup it not working properly and the crawler assumes it is named like "/sitemap.xml". ---
**Author:** ![Brian\_McGue](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brian_mcgue/32/63366_2.png) [@Brian\_McGue](https://discuss.elastic.co/u/Brian_McGue)\ **Post date:** [February 22, 2021, 6:06pm UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/11 "2021-02-22T18:06:51Z")
The proxy was a great choice! Well done. You are correct in that the crawler assumes the sitemap will exist at `example.com/sitemap.xml`. As noted in the documentation for [Crawler Configuration](https://swiftype.com/documentation/site-search/crawler-configuration/advanced-settings#sitemap-basics), you can configure a custom sitemap URL. Also, I agree, the messaging around recognizing a sitemap on the robots.txt could be improved if that sitemap URL does not match a domain that's been configured to crawl. ---
**Author:** ![Marcelo\_da\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelo_da_silva/32/77245_2.png) [@Marcelo\_da\_Silva](https://discuss.elastic.co/u/Marcelo_da_Silva)\ **Post date:** [February 23, 2021, 2:35pm UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/12 "2021-02-23T14:35:31Z")
Good! 🙂 Thanks @Brian_McGue and @Sean_Story for your support! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 23, 2021, 2:36pm UTC](https://discuss.elastic.co/t/sitemap-for-example-com-located-at-sub-example-com/262631/13 "2021-03-23T14:36:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.