Ghost Web Crawlers

Elastic Stack Elasticsearch
1

I am using Elastic, Kibana, and Enterprise Search 8.14 on Elastic Cloud. I was test crawling some domains with the "Elastic Web Crawler" with multiple custom crawl schedules (top right drop down Kibana). Later, I deleted these crawlers, but 3 new crawlers have appeared in the dashboard with the index name "--" which cannot be inspected or deleted as I cannot enter the name. They also have "Ingestion Status" set to incomplete. These are the only crawlers on the dashboard. Also, it lists I have 4 "running crawlers" listed on the dashboard. I cannot find these indices either.

Is there a method for me to stop/delete these indices/crawlers?

crawlers
crawlers1906×878 90.2 KB

2

Removed #elastic-cloud

3

Hey @mszal_ib! I confirm I could reproduce an issue in 8.14 and it seems that we have a bug here. We will address it.

For now you can use following steps to clean up you crawler overview page

Step 1: list your crawlers

GET _connector?service_type=elastic-crawler

Take the id from the response use to delete crawlers without index names:

DELETE _connector/{id}

Let me know if it fixes your issue.

4

I can confirm this was able to delete the crawlers & active crawlers. However, now I have many orphan crawls. Will these eventually clean up, or is there another method to delete these?

orphan_crawls
orphan_crawls783×195 7.78 KB

5

I was able to solve the orphaned crawler's issue by finding the sync jobs

GET _connector/_sync_job

and deleting them with

DELETE _connector/_sync_job/{id}

Thank you for the help @Jedr_Blaszyk.

1 Like