API Interface for Elastic Search Index

Elastic Stack Elasticsearch
1

Hi,

I am having some trouble with the API interface for the Elasticsearch web crawler.

Background:
I have a created an Elastic search index with a web crawler.

I have also created an engine using the "Elasticsearch index-based" engine type.

I then try to initial a crawl via a API call as documented here:

Without any additional configuration, the following error message is returned: No crawler domains configured on the engine [my-engine]

So, after this I run the following POST request:
Endpoint:
[my-elastic-instance]/api/as/v1/engines/[my-engine]/crawler/domains

Payload:
{ "name": "https://some.url.here", "auth": { "type": "basic", "username": "username", "password": "password" } }
Which gets me past the 'no domains configured' error message.

However, now when I try and start a crawl request via an API call, it seems to be stuck with the crawler status flip flopping between:

"status": "started"
and
"status": "suspended"

Has anyone else experienced this problem and does anyone know a way to get this to work?

Thank you kindly for your response

2

Hi @raylowe - can you clarify, are you using the Elastic Web Crawler, or the App Search Web Crawler? These are not the same, and can cause some confusion.

The Elastic Web Crawler does not have an API, which is why I'm asking. The link you referenced is for the App Search Web Crawler's API. So if you're using the Elastic Web Crawler, but trying to use the App Search API, that may explain why you're running into issues.

3

Hi @Sean_Story

Thanks for your quick reply. I am using the Elastic Web Crawler and your answer makes sense. Thanks for the clarification! Do you know if there are plans to provide an API for the Elastic Web Crawler in the future?

Cheers,

4

There are not plans to add APIs for the Elastic Web Crawler. Instead, we're investing in a third (4th, if you count swiftype crawler :see_no_evil: ) crawler, the Open Crawler: GitHub - elastic/crawler

We're building this one from a CLI up, instead of from the UI down, and we hope it'll be a much more developer-friendly experience in the long run. It is still in beta, but is under active development, so don't hesitate to give it a try and send us any feedback you have, either through these discuss forums or issues/PRs on that repo.

1 Like