What is a Robots.txt file?
Saturday, May 26th, 2007Robots.txt is commonly referred to as the robot exclusion protocol. This refers to the act of including a robots.txt file in a web directory, with the purpose of telling web crawlers and search engines not to index certain areas of a web site.
