Hello everybody,
I need help with the web crawler. I want the web crawler to use google and index all pages on a certain topic. Does it work and if so, how?
Thanks for help.
Hi Cama,
Unfortunately, no - the web crawler is only intended to crawl sites that belong to you and that you have control over (for changing things like noindex headers/meta tags, robots.txt, and so forth). It is not intended to be used for 3rd party sites or services such as Google.
Hope that helped,
Constance