Google offers robots.txt generator in Webmaster Tools
Google is offering webmasters a robots.txt generator in Google Webmaster Tools.
In case you don’t know what a robots.txt file is, it is a file which tells the Google searchbot which areas of the site to crawl and which areas NOT to crawl. It’s essential for all webmasters to have a robots.txt file in their website root directory and this robots.txt generator in Webmaster Tools makes the whole process much easier.
Related posts:
