Home / server / Questions / 171985
Accepted
qxotk
Asked: 2010-08-19 10:09:21 +0800 CST

How Can I Encourage Google to Read New robots.txt File?

  • 772

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update.

Is there any way I can encourage Google to re-read my robots.txt as soon as possible?

UPDATE: Under Site Configuration | Crawler Access | Test robots.txt:

Home Page Access shows:

Googlebot is blocked from http://my.example.com/

FYI: The robots.txt that Google last read looks like this:

User-agent: *
Allow: /<a page>
Allow: /<a folder>
Disallow: /

Have I shot myself in the foot, or will it eventually read: http:///robots.txt (as it did the last time it read it)?

Any ideas on what I need to do?

google robots.txt google-webmaster-tools

5 Answers

  1. Best Answer

    In case anyone else runs into this problem there is a way to force google-bot to re-download the robots.txt file.

    Go to Health -> Fetch as Google [1] and have it fetch /robots.txt

    That will re-download the file and google will also re-parse the file.

    [1] in the previous Google UI it was 'Diagnostics -> Fetch as GoogleBot'.

    • 25

  2. I know this is very old, but... If you uploaded the wrong robots.txt (disallowing all pages), you can try the following:

    • first correct your robots.txt to allow the correct pages, then
    • upload a sitemap.xml with your pages

    as google tries to read the xml sitemap, it will check it agains robots.txt, forcing google to re-read your robots.txt.

    • 4

  3. After have the same problem I sucessfuly made google reread my robots.txt file by submiting on this url:

    https://www.google.com/webmasters/tools/robots-testing-tool

    • 2

  4. OK. Here is what I did, and within a few hours, Google re-read my robots.txt files.

    We have 2 sites for every 1 site we run. Let's call them the canonical site (www.mysite.com) and the bare-domain site (mysite.com).

    We have our sites setup so that mysite.com always returns a 301 redirecting to the www.mysite.com.

    Once I setup both sites in Google Webmaster tools, told it that the www.mysite.com is the canonical site, it soon after read the robots.txt file on the canonical site.

    I don't really know why, but that's what happened.

    • 1

  5. Shorten google scan interval for some days.

    Also, I've seen there buttom to verify your robots.txt, this might force it to google, but I am not sure.

    • 0