Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
http://www.robotstxt.org/
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from ...
http://en.wikipedia.org/wiki/Robots.txt
A Standard for Robot Exclusion Table of contents: Status of this document Introduction Method Format Examples Example Code Author's Address Status of this document
http://www.robotstxt.org/orig.html
robots.txt generator designed by an SEO for public use. Includes tutorial.
http://www.mcanerin.com/EN/search-engine/robots-txt.asp
User-agent: * Disallow: /search. Disallow: /groups. Disallow: /images. Disallow: /catalogs. Disallow: /catalogues. Disallow: /news. Allow: /news/directory
http://www.google.com/robots.txt
Learn about the robots.txt, and how it can be used to control how search engines and crawlers do on your site.
http://www.javascriptkit.com/howto/robots.shtml
User-agent: * Crawl-delay: 10 . Sitemap: http://www.whitehouse.gov/feed/media/video-audio
http://www.whitehouse.gov/robots.txt
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a ...
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
Information on using the robots.txt file to keep web crawlers, spiders and robots from indexing certain sections of a site.
http://www.searchtools.com/robots/robots-txt.html
Information on the robots.txt and how it effects your website. Also includes a free robots.txt generator
http://www.robotstxt.ca/
# robots.txt for http://www.wikipedia.org/ and friends # # Please note: There are a lot of pages on this site, and there are # some misbehaved spiders out there that go _way_ ...
http://en.wikipedia.org/robots.txt
|
