FSCrawler over HTTPS

Elastic Stack Elasticsearch
1

Greetings:

I am trying to use FSCrawler to crawl a remote site over https. Does anyone know if this is even possible? I've read the docs and tried doing it over SSH butI get the "Error while crawling https://myserver.pages.com/pages/pages: https://myserver.pages.com/pages/pages doesn't exists."

We are trying to crawl some pages that exist on a public github repository.

2

FSCrawler has not been designed to crawl a website. https is not ssh protocol.

You might want to explore other solutions:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.