Home / server / Questions / 103856
In Process
Joannes Vermorel
Asked: 2010-01-19 09:12:16 +0800 CST

Make Google Apps site publicly accessible while disabling crawlers with robots.txt?

  • 772

I would like to create a publicly accessible Google Apps site (i.e. users do not need to be authenticated to access the content) while maintaining a policy crawlers and bots exclusion with Robots.txt. Does anyone know how to do that?

access-control-list g-suite robots.txt

1 Answers

  1. robots.txt doesn't prevent interactive browsers from using the site. It is only used by robots like crawlers, feedreaders, recursive download tools (though the latter will let the user override it).

    • 1