Add site to crawler via api

I'd like to add sites to search via the API which akaict isn't currently possible officially - is there :

  • a way to do this unofficially?
  • plans to do this eventually?

@hilt86 :wave:

Assuming you're talking about the web crawler beta released with 7.11 - we do have plans of extending the documented APIs to support the same functionality the web crawler offers in the dashboard.

As for unofficial ways to do this: you could figure out the private endpoints being used in the dashboard with your browser's developer tools and then if you're using a non-SAML auth source, you could perform a request with basic auth to that endpoint. This is obviously not recommended, and would most likely break in upcoming releases :smirk:

ok thanks - how does elastic app search work with the users and roles in Kibana? I want to create an account that can only manage app search and indexes created therein..

ok I managed to add 42000 sites to the crawler and it didn't really appreciate that...asides from the fact that the crawler ui doesn't paginate the real issue was that initiating a crawl started and then promptly died!

I need to index about ~50k+ sites - is crawler cut out for this?