 |  | |||||||
|
30 Mar This post is from from my other blog here Google has added yet another handy feature to the Google Webmaster Tools. Now website owners can easily create Robots.txt files and control which web pages search engines see and "don't" see on their sites. For those of you unfamiliar with Robots.txt files, Wikipedia explains them: release post highlights, not all robots obey the conventions of the Robots.txt protocol, but rest assured most of the major search engines do, which is where most of your website crawling will come from.
| ||||||||
 |  |
by 
