Home / server / Questions / 211395
Accepted
Wayne Conrad
Asked: 2010-12-11 10:25:50 +0800 CST

How to find the real IP to which IPVS is routing a virtual IP

  • 772

I'm trying to find a problem server hiding behind a virtual IP (using LVS/ipvs). I've got a test program that sends requests to the virtual IP until it gets the bad response, but how can I tell to which real IP a request to the virtual IP got routed?

On the box doing the virtual IP magic, here's the virtual IP configuration (for the service I care about):

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
...
TCP  10.1.0.254:5025 nq
  -> 10.1.0.5:5025                Route   1      0          1         
  -> 10.1.0.6:5025                Route   1      0          5         
  -> 10.1.0.7:5025                Route   1      0          2         
  -> 10.1.0.9:5025                Local   1      0          3         
  -> 10.1.0.11:5025               Route   1      0          3         
...

My client program is sending TCP requests to 10.1.0.254:5025, usually getting a good response but sometimes a bad response. With this few servers, I could send my request to each server in turn until I discover the culprit, but I wonder if that technique will scale as we add servers. What means exist for me to find out where requests got routed?

  • Kernel: Linux 2.6.32
  • OS: Debian testing (whatever that's called these days).
  • ipvsadm is version 1.25, compiled with ipvs v1.2.1
linux-networking lvs

3 Answers

  1. Best Answer

    This is quite a typical problem with IPVS/ldirectord, you have several options to try to resolve this

    • Check logging on the destination servers looking for the request (not very accurate but most of the time it'll cover the bill)
    • If your destination server replies with headers (like apache or ftp) just add a new custom header like X-Served-by or some hint in the welcome header
    • Activate logging on ldirectord, the logging is a bit obscure but I'm pretty sure it'll be useful, just add to your ldirectord.cf logfile="/var/log/ldirectord.log" in the global options
    • 1

  2. ipvsadm -Lcn on the LB (aka "director") should list your connections. Grep for the IP you are searching.

    • 1

  3. You best bet would be to rn tcpdump on the real servers to see where you request is going.

    Gotcha. LVS-TUN listen on the on the virtual interface for the client address because the packets are encapsulated you wouldn't be able to identify the packet based on users IP

    $ tcpdump -i tunl0 host USER_IP

    • 0