bot-throttle

regulate the crawl rate of a search engine bot by using a bandwidth throttle on the web-server

We might need to bot-throttle googlebot this month because we're hit the bandwidth limit on our shared server.

by Arunabh Das Sharma June 17, 2008