Home / server / Questions / 44110
Accepted
Petrogad
Asked: 2009-07-23 07:40:03 +0800 CST

Is there a way to tell robots not to look at a directory?

  • 772

I have a directory full of images, that I do not want viewed by robots, is there a way to make this directory private?

Thanks!

seo

7 Answers

  1. Best Answer

    How to Use Robots.txt in Search Engine Optimization ( SEO )

    Type-in " Disallow: /folder name "
    for any folder containing files you don’t want search engines to crawl.

    Or, type-in " Disallow: /filename.filetype "
    for any page or file you don’t want search engines to index.

    Note of the following example :

    * Disallow: /private/
* Disallow: /private/image01.jpg
    • 10

  2. Create a robots.txt file in your site's root directory. Inside that file put

    User-agent: *
Disallow: /images
    • 1

  3. It's called a robots.txt file, and most all robots will honor your requests in your robots.txt.

    You'll want something like the following:

    User-Agent: *
Disallow: /path/to/images/
    • 1

  4. You want robots.txt

    • 0

  5. Yeah like this:

    User-agent: *
Disallow: /MyImageDirectory
    • 0

  6. In your root directory you can place a robots.txt file that forbids robots to index specific directories.

    • 0

  7. robots.txt is only obeyed by well-behaved robots. not all of them are well-behaved like those of the major search engines - some are written by incompetent buffoons, some are written by spammers trawling for email addresses, and so on.

    anyway, you also want to turn Indexes off for that directory...either with a .htaccess file or in your apache conf like so: 

    <Directory /path/to/images/dir>
  Options -Indexes
</Directory >

    or by by placing a blank index.html file in there.


    this will stop them from seeing a directory listing of the image files. it won't stop them from following direct links to the images from other pages on your site.

    there's actually very little you can do to really prevent robots from accessing public files on your webserver that won't either inconvenience your users or increase the load on your server by only serving images through a script that only serves to logged-in users (e.g. by checking a cookie).

    • 0