View Poll Results: Which search engines do you use the most?

46. You may not vote on this poll
  • Google

    41 89.13%
  • Alltheweb

    0 0%
  • Altavista

    1 2.17%
  • Lycos

    0 0%
  • Teoma

    0 0%
  • HotBot

    0 0%
  • Yahoo!

    3 6.52%
  • Ask Jeeves

    1 2.17%
Page 2 of 2 FirstFirst 12
Results 11 to 13 of 13
  1. #11
    Join Date
    May 2003
    Logan, UT


    robots.txt is a file that tells spiders sent out from search engines which content they can index and what is allowed or disallowed. For instance in my robots.txt file I could allow every search engine except Google, or I could only allow google and disallow every other search engine. I could also restrict access to certain folders on my site for the spiders. Here is an example of a robots.txt file. This file is allowing all spiders (*) to index the site and disallowing them to index the cgi-bin directory.

    User-agent: *
    Disallow: /cgi-bin/

  2. #12
    Junior Member
    Join Date
    Oct 2003


    Thanks for your help
    Just a new customer

  3. #13
    Moderator wildjokerdesign's Avatar
    Join Date
    Jun 2003
    Kansas City Mo


    Just thought I would add that you place the robots.txt in your home directory where you have your home page i.e. /var/www/html/
    Please remember your charity of choice:

    Handy Links: | Plain Text Editors: EditPlus | Crimson

Similar Threads

  1. Need help with re-direct and search engine
    By thespiritedwoman in forum Search Engines
    Replies: 8
    Last Post: 02-05-2010, 07:34 AM
  2. Search Engine Tools - 3 Months Free
    By WestHost - BErickson in forum Search Engines
    Replies: 2
    Last Post: 02-03-2010, 11:30 PM
  3. mod_rewrite and search engine url's with addon domain
    By erim in forum General Discussion
    Replies: 7
    Last Post: 02-16-2008, 05:08 PM
  4. Moving directory. How to redirect existing search engine links?
    By davidg_it in forum General Discussion
    Replies: 1
    Last Post: 09-14-2005, 08:29 AM
  5. Search Engine submit question
    By lukegt in forum General Discussion
    Replies: 2
    Last Post: 12-28-2003, 07:27 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts