# Can I crawl intranet web?

**URL:** <https://discuss.elastic.co/t/can-i-crawl-intranet-web/278684>\
**Category:** Elastic Search\
**Created:** [July 14, 2021, 2:53pm UTC](https://discuss.elastic.co/t/can-i-crawl-intranet-web/278684 "2021-07-14T14:53:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sguerrero](https://avatars.discourse-cdn.com/v4/letter/s/87869e/32.png) [@sguerrero](https://discuss.elastic.co/u/sguerrero)\
**Post date:** [July 14, 2021, 2:53pm UTC](https://discuss.elastic.co/t/can-i-crawl-intranet-web/278684/1 "2021-07-14T14:53:01Z")

</div>

I can crawl external webs but when I tried to crawl a web that is only accessible from my internal net I couldn't do.  
I far as I've read it seems feasible, right? I've search the web but I couldn't find any post/topic talking about this.  
Thanks!

---

<div class="post-metadata">

**Author:** ![oleksiy-elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiy-elastic/32/49074_2.png) [@oleksiy-elastic](https://discuss.elastic.co/u/oleksiy-elastic)\
**Post date:** [July 14, 2021, 3:28pm UTC](https://discuss.elastic.co/t/can-i-crawl-intranet-web/278684/2 "2021-07-14T15:28:25Z")

</div>

Hello,

By default, Enterprise Search crawler configuration prohibits access to any websites that use a private IP address (or a loopback IP). This is by design, because it opens up your deployment to a number of pretty serious attacks including [SSRF](https://owasp.org/www-community/attacks/Server_Side_Request_Forgery). On Elastic Cloud, this configuration is impossible to change since we have to protect our internal services from SSRF attacks.

If you're running Enterprise Search in your own environment (self-managed) and you both completely trust all users with access to its management console and you own (control DNS configuration) the domains you're planning to crawl, you can disable this protection by setting the `crawler.security.dns.allow_private_networks_access` to `true` in your `enterprise-search.yml` and restarting the service. But, once again, I'd like to point out that it opens you up to a number of different attacks and you need to be very careful with what you crawl with this setting enabled.

I hope this helps.

---

<div class="post-metadata">

**Author:** ![sguerrero](https://avatars.discourse-cdn.com/v4/letter/s/87869e/32.png) [@sguerrero](https://discuss.elastic.co/u/sguerrero)\
**Post date:** [July 14, 2021, 4:20pm UTC](https://discuss.elastic.co/t/can-i-crawl-intranet-web/278684/3 "2021-07-14T16:20:50Z")

</div>

Thank you very much, you've answered my question!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 8:31am UTC](https://discuss.elastic.co/t/can-i-crawl-intranet-web/278684/4 "2022-11-04T08:31:28Z")

</div>


