Thanks for your response @nfeekery
Yes, I had seen this solution with Crawl-delay rule in the robots.txt file. But the problem is some websites may not have the robots.txt file or at least may not have the Crawl-delay option in the robots.txt file. I was looking for a more general option...