12 Do's and Don'ts for a Successful data integration
http://remingtonmoeg469.yousher.com/data-intergration-expectations-vs-reality
This allows Goggle (person-agent title GoogleBot) to go to every web page and directory, when at the same time banning Check with Jeeves with the web page absolutely