Home / server / Questions / 495603
In Process
Abraham Vegh
Asked: 2013-04-03 18:58:17 +0800 CST

Why is Googlebot unable to reach URLs that are valid and reachable?

  • 772

For a while, I had a problem with Googlebot being unable to crawl my site (and in fact I was having this problem for so long that my site was actually de-listed from Google).

Testing my site in Google Webmaster Tools with Fetch As Googlebot actually returned a blank page: Googlebot was unable to reach my site, and could provide no diagnostic message that could explain the lack of reachability.

What’s up with that?

linux

1 Answers

  1. As it turns out, for reasons still unknown, my correctly-configured networking service on my web server was bringing up its IP address with a /8 netmask, instead of the (in my case) correct netmask of /24.

    Manually removing the offending entry from my server’s routing table and adding the correct entry resolved the issue, and Googlebot is now happily visiting my web server again:

    root@web:/# ip addr show eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.0.2.7/8 brd 192.255.255.255 scope global eth0
root@web:/# ip addr add 192.0.2.7/24 dev eth0
root@web:/# ip addr del 192.0.2.7/8 dev eth0
root@web:/# ip addr show eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.0.2.7/24 scope global eth0

    (IP addresses changed to protect the innocent. Adding the correct entry before removing the incorrect one helps keep your SSH session open if you’re connected via that address.)

    • 1