Top Definition
File that stops search engines from crawling a specified page or directory.
Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
by KFM December 23, 2004
Free Daily Email

Type your email address below to get our free Urban Word of the Day every morning!

Emails are sent from We'll never spam you.