Home / server / Questions / 422204
Accepted
George Tasioulis
Asked: 2012-08-29 14:27:25 +0800 CST

Secondary IP (eth0:0) acts like main server IP

  • 772

I have a CentOS server, configured with 4 consecutive IPs:
eth0 5.x.x.251
eth0:0 5.x.x.252
eth0:1 5.x.x.253
eth0:2 5.x.x.254

The problem is that all traffic goes out to the internet with eth0:0 (5.x.x.252) as the source IP, instead of eth0.
# curl ifconfig.me
5.x.x.252

How can I fix this, so that all traffic goes out via eth0, ie my main IP?

PS: My server is VPS running on a Xen dom0, the latter being configured in routed mode networking.

Thanks in advance!

Server configuration 

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.251  Bcast:5.x.x.255  Mask:255.255.255.255
          inet6 addr: fe80::x:x:x:x/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14675569 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9463227 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4122016502 (3.8 GiB)  TX bytes:25959110751 (24.1 GiB)
          Interrupt:23 

eth0:0    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE   
          inet addr:5.x.x.252  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23 

eth0:1    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.253  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23 

eth0:2    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.254  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23  

# cat /etc/hosts
127.0.0.1       localhost.localdomain localhost
5.x.x.251       [fqdn] [hostname]

# cat ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=5.x.x.251
NETMASK=255.255.255.224
SCOPE="peer 5.x.y.82"

# cat ifcfg-eth0:0
DEVICE=eth0:0
BOOTPROTO=static
ONBOOT=yes
IPADDR=5.x.x.252
NETMASK=255.255.255.224

# cat route-eth0 
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=5.x.y.82

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
5.x.y.82        0.0.0.0         255.255.255.255 UH        0 0          0 eth0
5.x.x.224       0.0.0.0         255.255.255.224 U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
0.0.0.0         5.x.y.82        0.0.0.0         UG        0 0          0 eth0

Update 29 August 2012
When I run /etc/init.d/network restart I get an RTNETLINK answers: File exists error, which leads me to believe that there's another route to another network using the default gateway, as seen here.

# /etc/init.d/network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:  RTNETLINK answers: File exists
                                                           [  OK  ]

The sure thing is that when I first delete ip route del default via 5.x.y.82 dev eth0 and then add it again with the .251 IP, everything works as expected.
Is there any way to add a route del command either to the ifcfg-eth0 or the route-eth0 files? I can see that this can be done with pre-up ip addr del ... on Debian (more info), but what about CentOS?

linux

7 Answers

  1. Best Answer

    From this document it appears that at least on CentOS 5 and I suspect above you can specify your routes by simply giving the correct iproute2 command arguments. (see the section on the IP Command Arguments Format)

    So instead of writing something like:

    # route-eth0 
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=5.x.y.82

    You could just have a file like this:

    default via 5.x.y.82 dev eth0  src 5.x.x.251

    I don't have a Redhat/Redhat derived box handy to test though.

    • 6

  2. First clear your 0.0.0.0 default gateway

    route del -net default

    Then declare default eth0 as default gateway device

    route add -net default gw 5.x.y.82 dev eth0

    It should work. Verify with

    ip ro li

    To save that as a rule try this messy thing:

    Open /etc/sysconfig/network-scripts/network-functions, locate function add_default_route () and in that function, after the call to find_gateway_dev add a line with GATEWAYDEV="eth0" .

    What happens is that the network-functions script decides which device gets to be the default gateway. It examines the route to the gateway IP with ip get route to GATEWAY and, using sed, matches the a device. It seems that eth0:0 gets to be the one each time, so by hardcoding GATEWAYDEV="eth0" you make sure that eth0 is always selected as the gateway device.

    In older versions of RH you could just edit /etc/sysconfig/static-routes and add default via 5.x.y.82 dev eth0

    As a last resort /etc/rc.local is always your friend!

    • 2

  3. If the aliased IP addresses are not to be used as source addresses to non-local destinations, they should not be in the same subnet as the target of your default route. So change their netmasks to 255.255.255.255 and remove their broadcast addresses.

    • 2

  4. Sounds like you might be a victim of a setting in newer kernels. Enabling alias promotion will get past that.

    Set this in your /etc/sysctl.conf

    net.ipv4.conf.all.promote_secondaries = 1

    Incidentally, do you have any funky SNAT/MASQUERADE iptables rules that are altering the outbound source IP?

    • 2

  5. Since they are all on the same subnet the kernel will pick one to use.. You can verify this by running

     ip addr list dev eth0

    You have to do some iptables magic to get it to work correctly. See the following

    When IP aliasing how does the OS determine which IP address will be used as source for outbound TCP/IP connections?

    • 1

  6. route and ifconfig use old, deprecated, network interfaces in kernel that don't know about policy based routing. You should be using the ip command, especially for routing-related information.

    That being said, ip ro li or the full command ip route list table main, outputs the actual routing table used by the kernel. ip rule list will show you all the routing tables in use (there should at least be local, main and default).

    As you've posted (from http://pastebin.com/A1KRiWTz):

    # ip ro li
5.x.y.82 dev eth0  proto kernel  scope link  src 5.x.x.251 
5.x.x.224/27 dev eth0  proto kernel  scope link  src 5.x.x.252 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 5.x.y.82 dev eth0  src 5.x.x.252

    you need to remove the default route and recreate it with correct source address:

    ip route del default 
ip route add default via 2.x.y.82 dev eth0 src 5.x.x.251 scope global

    I don't know how to do it the RedHat way though (where to put it in RedHat-specific config files), using /etc/rc.local should work, albeit it's somewhat hacky solution. You could try using system-config-network

    • 0

  7. You need to write a PREROUTING rule for inbound and outbound traffic in iptable, you need to say any tcp or UDP traffic or port 80 or 8080 (for default internet only) should go to desired eth.

    • -3