I am running a CentOS 6.2 server as my gateway and firewall, while also providing some internal services. This is a setup I have had for years using various hardware and distributions (redhat based). Recently I have ran across an internet connectivity issue and I believe it is due to either a flaw with my ISP (Roadrunner, upstate new york) or a flaw with my configuration (default) for dhclient.
I am not using NetworkManager on this server, as the network configuration is static and the server runs 24/7 as a gateway. I have my sysconfig network-script interface configuration as below:
which has configured the interface on boot and to use DHCP, via the dhclient utility. I have a valid iptables firewall script that I have been continually working on for years in place to provide routing/NAT functionality, but that is irrelevant to my problem.
For the last week or two (at least, I've seen these log entries on and off for a while now) I see that once DHCP lease offered from my ISP reaches the halfway point, triggering a renewal, dhclient enters a loop where every 15 seconds it issues a unicast DHCPREQUEST to the DHCP Server entry specified in /var/lib/dhclient/dhclient-eth1.leases file (see below). This continues for hours until either network connectivity breaks or it eventually attempts a broadcast discover and receives a new lease properly.
The dhclient request loop is always unicast, always using the DHCP Server address specified in the lease it is attempting to renew, and it always uses the same xid value for each of these requests. I am wondering, is there a way to force dhclient to always issue a broadcast DHCPDISCOVER instead of the unicast REQUEST packet for renewal? Is there a possible configuration problem or is this just Time Warner's flaky DHCP Service? I have used TWC as my ISP for the last 5 years and never had this issue when using Linux as a gateway.
Here is my interface configuration script:
/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
NAME=Internet
HWADDR=00:D0:B7:**:**:**
MTU=1500
BOOTPROTO=dhcp
PEERDNS=no
IPV6INIT=no
ONBOOT=yes
Here is an example dhclient-eth1.leases file (current, but will start the loop in ~6-8 hours)
lease {
interface "eth1";
fixed-address 74.***.***.***;
option subnet-mask 255.255.240.0;
option routers 74.***.***.***;
option dhcp-lease-time 43200;
option dhcp-message-type 5;
option domain-name-servers 209.18.47.61,209.18.47.62;
option dhcp-server-identifier 10.111.64.1;
option interface-mtu 576;
option broadcast-address 255.255.255.255;
option domain-name "rochester.rr.com";
renew 3 2012/01/18 21:51:02;
rebind 4 2012/01/19 02:57:58;
expire 4 2012/01/19 04:27:58;
}
and an excerpt from /var/log/messages regarding this issue (started at about 12:30AM and continues till 11:30AM this morning :
... Long listing of DHCPREQUEST nearly identical to below
Jan 17 16:50:59 server dhclient[1384]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x54a5b374)
Jan 17 16:51:13 server dhclient[1384]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x54a5b374)
Jan 17 16:51:21 server dhclient[1384]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x54a5b374)
Jan 17 16:51:31 server dhclient[1384]: DHCPREQUEST on eth1 to 255.255.255.255 port 67 (xid=0x54a5b374)
Jan 17 16:51:31 server dhclient[1384]: DHCPACK from 10.111.64.1 (xid=0x54a5b374)
Jan 17 16:51:31 server dhclient[1384]: bound to 74.69.54.153 -- renewal in 17309 seconds.
It seems here to have successfully, finally, got an DHCPACK after a LONG list of attempts
Last evening at around 7:30PM, well past the log entry above at 16:51 and evenutally rebooted the server for other reasons, which causes the REQUEST line below.
Jan 17 20:11:51 server dhclient[3872]: DHCPREQUEST on eth1 to 255.255.255.255 port 67 (xid=0x4a4507ce)
Jan 17 20:11:51 server dhclient[3872]: DHCPACK from 10.111.64.1 (xid=0x4a4507ce)
Jan 17 20:11:51 server dhclient[3872]: bound to 74.69.54.153 -- renewal in 17073 seconds.
Jan 18 00:56:24 server dhclient[3917]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x4a4507ce)
Jan 18 00:56:32 server dhclient[3917]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x4a4507ce)
Jan 18 00:56:46 server dhclient[3917]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x4a4507ce)
Jan 18 00:57:04 server dhclient[3917]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x4a4507ce)
Jan 18 00:57:24 server dhclient[3917]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x4a4507ce)
.... omission of several hours and many many lines of the above, every ~15 seconds This is where i manually brought the interface down and up.
Jan 18 11:27:29 server dhclient[3917]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x4a4507ce)
Jan 18 11:27:45 server dhclient[3917]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x4a4507ce)
Jan 18 11:27:52 server dhclient[3917]: DHCPREQUEST on eth1 to 10.111.64.1 port 67 (xid=0x4a4507ce)
Jan 18 11:27:58 server dhclient[16174]: DHCPREQUEST on eth1 to 255.255.255.255 port 67 (xid=0x63741216)
Jan 18 11:27:58 server dhclient[16174]: DHCPACK from 10.111.64.1 (xid=0x63741216)
Jan 18 11:27:58 server dhclient[16174]: bound to 74.69.54.153 -- renewal in 19384 seconds.
Ive always had some MTU issues regarding fragmentation with my firewall, but that does not seem to be the root cause here, and would be a separate question if any.
I had the same problem with the same ISP (RoadRunner). It looks like RR provides an invalid or unreachable dhcp-server-identifier host IP. While it would be nice if the ISP fixed the problem, you can add the following to your
/etc/dhcp/dhclient.conf(location may be different in your distribution):This will cause the client to ignore the IP address of the dhcp server provided in the response and send a broadcast for the DHCPREQUEST. This is a hack. It probably violates the governing RFCs but it works for me.
I have this problem with my cable provider. They do not respond to unicast DHCPREQUEST packets. I use:
iptables -t nat -A OUTPUT -d 10.0.0.0/255.0.0.0 -o eth1 -p udp -m udp --dport 67 -j DNAT --to-destination 255.255.255.255
to turn these into broadcasts, and the problem goes away.
It is most likely due to odd firewall rules or some weird configuration on the service provider end regarding not allowing directed DHCP requests. Your DHCP client is most likely behaving properly.
When the client reaches the RENEW time, it'll send back unicast DHCPREQUESTS to try and renew the least.
When it reaches the EXPIRE time, then it'll start broadcasting again. And that's what we're seeing from the logs you pasted.
Doing something kludgy such as killing dhclient, scrubbing the lease file and restarting dhclient may make things work. But really shouldn't be required.
Do you have logs from when network connectivity actually breaks?
I had this too, the problem is that dhclient is sending unicast (server IP) requests to the ISP and my ISP does not accept unicast only multicast (255.255.255.255), dhclient should fall back to multicast when the unicast fails but it does not seem to do that until rebind or expire (not sure which) and that can be days later.
My solution was to run this script daily with root from /etc/crontab
The first line kills off the existing dhclient that is sending the unicast requests. The second line starts a new instance that uses multicast.
make the script executable
add a line like this to /etc/crontab
I had also DHCPREQUEST spam on my logs (Debian 8). I had dhclient (isc-dhcp-client 4.3.1). My setup was a bit peculiar, but I fixed my problems just by removing isc-dhcp-client and installing dhcpcd 6.0.5 (by Roy Marples). dhcpcd seems to be smarter than isc-dhcp-client. Works like a charm now.