# Add site to crawler via api **URL:** \ **Category:** Elastic Search\ **Tags:** elastic-app-search\ **Created:** [February 14, 2021, 10:33pm UTC](https://discuss.elastic.co/t/add-site-to-crawler-via-api/264257 "2021-02-14T22:33:12Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [February 14, 2021, 10:33pm UTC](https://discuss.elastic.co/t/add-site-to-crawler-via-api/264257/1 "2021-02-14T22:33:12Z")
I'd like to add sites to search via the API which akaict isn't currently possible officially - is there : - a way to do this unofficially? - plans to do this eventually? ---
**Author:** ![orhantoy](https://avatars.discourse-cdn.com/v4/letter/o/c5a1d2/32.png) [@orhantoy](https://discuss.elastic.co/u/orhantoy)\ **Post date:** [February 14, 2021, 11:10pm UTC](https://discuss.elastic.co/t/add-site-to-crawler-via-api/264257/2 "2021-02-14T23:10:55Z")
@hilt86 👋 Assuming you're talking about the web crawler beta released with 7.11 - we do have plans of extending the documented APIs to support the same functionality the web crawler offers in the dashboard. As for unofficial ways to do this: you could figure out the private endpoints being used in the dashboard with your browser's developer tools and then if you're using a non-SAML auth source, you could perform a request with basic auth to that endpoint. This is obviously not recommended, and would most likely break in upcoming releases 😏 ---
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [February 16, 2021, 3:36am UTC](https://discuss.elastic.co/t/add-site-to-crawler-via-api/264257/3 "2021-02-16T03:36:19Z")
ok thanks - how does elastic app search work with the users and roles in Kibana? I want to create an account that can only manage app search and indexes created therein.. ---
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [February 17, 2021, 6:18am UTC](https://discuss.elastic.co/t/add-site-to-crawler-via-api/264257/4 "2021-02-17T06:18:45Z")
ok I managed to add 42000 sites to the crawler and it didn't really appreciate that...asides from the fact that the crawler ui doesn't paginate the real issue was that initiating a crawl started and then promptly died! I need to index about ~50k+ sites - is crawler cut out for this? ---
**Author:** ![Rich\_Kuzsma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rich_kuzsma/32/66159_2.png) [@Rich\_Kuzsma](https://discuss.elastic.co/u/Rich_Kuzsma)\ **Post date:** [February 25, 2021, 10:26pm UTC](https://discuss.elastic.co/t/add-site-to-crawler-via-api/264257/5 "2021-02-25T22:26:53Z")
Can you clarify: Are you trying to index 50K different sites (website domains), or 50K different pages within one website? The crawler should be able to crawl 50K pages on a website, but you'll need to increase these settings in `enterprise-search.yml`: ```auto #crawler.crawl.max_unique_url_count.limit: 10000 #crawler.crawl.max_duration.limit: 86400 # seconds ``` As for 50K _sites_... This is an interesting use case! I'd love to learn more about it. I don't know any reason why it _wouldn't_ work, but I have not tried this many. > initiating a crawl started and then promptly died! Could you share any error messages from the application logs under `log/crawler.log`? Have you configured any crawl rules for any sites? If the crawl rules reject the entry point URL itself, the crawler will start and stop immediately, thinking it has nothing to do. ---
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [March 1, 2021, 1:18am UTC](https://discuss.elastic.co/t/add-site-to-crawler-via-api/264257/6 "2021-03-01T01:18:21Z")
ok I will try that! I'm trying to index 50k different sites for an OSINT exercise for internal consumption. For example I want to index a bunch of sites and get notified if certain snippets of text appear on these sites (which I will use the API once I have a quality index) H ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 29, 2021, 1:18am UTC](https://discuss.elastic.co/t/add-site-to-crawler-via-api/264257/7 "2021-03-29T01:18:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.