# Crawler always fails when there are several domains

**URL:** <https://discuss.elastic.co/t/crawler-always-fails-when-there-are-several-domains/289734>\
**Category:** Elastic Search\
**Tags:** elastic-app-search\
**Created:** [November 20, 2021, 4:49am UTC](https://discuss.elastic.co/t/crawler-always-fails-when-there-are-several-domains/289734 "2021-11-20T04:49:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mbil](https://avatars.discourse-cdn.com/v4/letter/m/e8c25b/32.png) [@mbil](https://discuss.elastic.co/u/mbil)\
**Post date:** [November 20, 2021, 4:49am UTC](https://discuss.elastic.co/t/crawler-always-fails-when-there-are-several-domains/289734/1 "2021-11-20T04:49:45Z")

</div>

When I run the crawler with only one domain configured, it runs successfully, even this domain has thousands of pages.

After that, I delete the domain, add a new one and start the crawler again.

This works fine, but it's a very manual and unrealistic process.

So if I add multiple domains (the same ones that worked before), the crawler always fails. What is the reason? The crawler should work the same for one or multiple domains.

---

<div class="post-metadata">

**Author:** ![Felix\_Roessel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felix_roessel/32/41623_2.png) [@Felix\_Roessel](https://discuss.elastic.co/u/Felix_Roessel)\
**Post date:** [November 21, 2021, 11:13am UTC](https://discuss.elastic.co/t/crawler-always-fails-when-there-are-several-domains/289734/2 "2021-11-21T11:13:24Z")

</div>

Did you checked the logs?  
This Kibana Crawler Dashboard can help you.

> **[Download Web Crawler dashboard at elastic content share](https://elastic-content-share.eu/downloads/crawler-dashboard/)**
>
> A Kibana dashboard that visualizes the result of the crawler from Elastic Enterprise Search

---

<div class="post-metadata">

**Author:** ![Carlos\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carlos_d/32/126245_2.png) [@Carlos\_D](https://discuss.elastic.co/u/Carlos_D)\
**Post date:** [November 22, 2021, 11:59am UTC](https://discuss.elastic.co/t/crawler-always-fails-when-there-are-several-domains/289734/3 "2021-11-22T11:59:30Z")

</div>

Hi @mbil !

I've been able to setup crawling multiple domains. Is there any specific error you're retrieving from the application, either in the UI or logs?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 20, 2021, 11:59am UTC](https://discuss.elastic.co/t/crawler-always-fails-when-there-are-several-domains/289734/4 "2021-12-20T11:59:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.