Hello all-

Do search engine spiders crawl directories outside of the "www" folder? Does an .htaccess file keep robots out of a directory or do I need to add directories in my root to my robots.txt?

On a related issue. I am running a bulletin board on my VDS. I've noticed Googlebot crawling it a lot lately. Is there something additional that I need to do to protect my sql database so that robots don't index restricted information?

Also is it possible to add a specific bulletin board category or thread to the robots.txt file so that spiders index only selected categories or threads?

Thanks for your replies.