Do search engine spiders crawl directories outside of the "www" folder? Does an .htaccess file keep robots out of a directory or do I need to add directories in my root to my robots.txt?
On a related issue. I am running a bulletin board on my VDS. I've noticed Googlebot crawling it a lot lately. Is there something additional that I need to do to protect my sql database so that robots don't index restricted information?
Also is it possible to add a specific bulletin board category or thread to the robots.txt file so that spiders index only selected categories or threads?
Thanks for your replies.
Results 1 to 3 of 3
Thread: spiders .htaccess & robots.txt
11-05-2003, 01:33 PM #1
spiders .htaccess & robots.txt"Beware of all enterprises that require new clothes." -Henry David Thoreau
By ryanz in forum General DiscussionReplies: 2Last Post: 03-02-2010, 12:54 AM
By bossbn in forum General DiscussionReplies: 5Last Post: 11-27-2007, 06:31 PM
By extexas in forum General DiscussionReplies: 1Last Post: 08-09-2005, 02:19 PM
By zestgourmet in forum CGI Scripts / PerlReplies: 1Last Post: 03-08-2005, 05:58 PM
By foeggy in forum PHP / MySQLReplies: 6Last Post: 02-12-2004, 01:51 AM