Do search engine spiders crawl directories outside of the "www" folder? Does an .htaccess file keep robots out of a directory or do I need to add directories in my root to my robots.txt?
On a related issue. I am running a bulletin board on my VDS. I've noticed Googlebot crawling it a lot lately. Is there something additional that I need to do to protect my sql database so that robots don't index restricted information?
Also is it possible to add a specific bulletin board category or thread to the robots.txt file so that spiders index only selected categories or threads?
Thanks for your replies.