we run ECE on premise in our data center. We have deployed an Enterprise Search App Search engine. We want to use the web crawler functionality. But the connection towards public internet pages runs via a corporate proxy.
I could not find this in the documentation nor on this forum.
Error message in logs: "Allow none because robots.txt responded with status 599".
How can we configure the web crawler to use a proxy for internet connectivity?