No documents after 20 hours of crawling

[Hiren_Patel]

I've initiated a manual crawl 20 hours ago however I see no documents in the control panel.

I see a "Swiftype is currently indexing content for domains in this search engine. Pages will become available as they are indexed." message, but I suspect there's some issue that I'm not seeing.

I have a valid sitemap generated based on the guides provided, and I can see Swiftbot hitting our servers.

Is there a problem, because I don't see what I'm doing wrong.

We're on the Pro Plan and the engine key is: HApht89NAVsMfYA37u4p

Kind regards,
Hiren

[goodroot (Kellen Evan)]

Hey Hiren_Patel!

Welcome to the community!

It seems as though visiting your website generates a 401 error.

The crawler would see the same, and then give up.

Is there a way to address this error?

Kellen

[Hiren_Patel]

Hi @goodroot,

We have a rule in place to allow Swiftbot access. I can see the bot hitting our servers successfully - see below a snippet from the access log:

You can see they all show up with 200 status codes, meaning the bot is seeing the pages.

Thanks,
Hiren

[Hiren_Patel]

Just to update, it's now been five days since the crawler started. Still no documents.

I've even gone through and removed the basic authentication in hopes of resolving this but still no success.

[goodroot (Kellen Evan)]

@Hiren_Patel

Something is definitely funky.

Please email support, they'll be able to do a proper investigation.

Sorry for your troubles!

Kellen

[Hiren_Patel]

@goodroot

Yep. Can't figure it out.

I've actually emailed them on the same day I posted this (five days ago) here. I haven't heard anything back from them.

[goodroot (Kellen Evan)]

@Hiren_Patel !!

It looks like a response was sent to you on May 3rd at around 10AM PT.

Perhaps it went into spam?

I've added this post into the case notes. If you can find the email, responding to it will get the thread kicked up again.

Kellen