Home / server / Questions / 285447
In Process
Jesse Bunch
Asked: 2011-06-30 10:42:23 +0800 CST

Blocking GoogleBot in the Apache Configuration File

  • 772

For my staging server, I'm trying to figure out a way to block search engine bots entirely, rather than an individual .htaccess file or robots.txt file. The idea is that it's out of sight and out of mind when creating a new site on the staging server. Is there a way to detect the bot's user agent using the apache module and block that connection on a sever-wide level?

Thanks!

linux apache-2.2 .htaccess

1 Answers

  1. My suggestion would be to block everything OTHER than a known-good testing agent string. That way you block bots you've never heard of. You could also block all but a known good set of IPs with %{REMOTE_ADDR}.

    RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} !YourTestingAgent
RewriteRule (.*) /404.html
    • 5