Chul-Woong Yang's questions

I'm trying to setup transparent proxy network using container (docker)

Client (C)         Proxy (P1)      Proxy (P2)
10.10.1.1/24      10.10.2.1/24    10.10.3.1/24
    veth0             veth0          veth0
     |                 |               |
  veth pair         veth pair      veth pair
     |                 |               |
 -----------(HOST)----------------------------
client-veth0       p1-veth0          p2-veth0
10.10.1.2/24      10.10.2.2/24     10.10.3.2/24
     |                 |               |    172.16.202.30
     +-----------------+---------------+------- enp4s0 ---- INTERNET

I set up this network in Linux netns environment by running configuration commands below inside each namespace, after setting up host portion:

• ip netns exec {target_ns} ip addr...
• ip netns exec {target_ns} ip link...
• ip netns exec {target_ns} ip route ...

Docker does networking by default and connects all containers with bridged networking. I want to setup separate data-plane, leaving management network (docker network by default) aside. So I have to make additional veth pair, put one end to each containers, and do policy routing on newly established veth interfacess.

For this I made a runscript: https://gist.github.com/cwyang/23220d7fed5a0cc9af21949aad70e2f6.

I want to set up corresponding network in Docker-compose environment.

Should I make a runscript again and run a series of docker exec instead of ip netns exec, to set up a network inside container, after running three containers?

If not, what's the recommended way?

In other words, when one want to do some custom-setup from host to docker container, is there plan B? (A) Docker(-compose) up and user does a series of docker execs (B) Docker up with some configuration to run after the container starts up. Configuration is run by docker, not by user.

When a container is down for a reason and docker(-compose) relaunches it, the configuration should be applied. So if the mechanism is provided by infra, users need not monitor each participating container to re-apply the configuration by themselves. So I wonder there is plan B.

Any help will be appreciated deeply. Thank you.

I'm trying to setup transparent proxying networks on my host.

Real Client and Proxy targets are containters but in this experiment I use netns (network namespace) separated envinroment.

To redirect client traffic to proxy transparently, I use policy routing.

 Client (C)         Proxy (P)
 10.10.1.1/24      10.10.2.1/24
     veth0             veth0
      |                 |
   veth pair         veth pair
      |                 |
  -----------(HOST)--------------
 client-veth0       proxy-veth0
 10.10.1.2/24      10.10.2.2/24
      |                 |            172.16.202.30
      +-----------------+-------------- enp4s0 ---- INTERNET

# Policy Routing on Host
# [Client->Proxy]
# ip rule:  from 10.10.1.0/24 iif client-veth0 lookup 100
# ip route: (100) default via 10.10.2.1 dev proxy-veth0
# [Proxy->Internet]
# ip route: (master) default via 172.16.202.1 dev enp4s0 proto static metric 100
# iptables: -t nat -A POSTROUTING -s 10.10.1.1/32 -o enp4s0 -j MASQUERADE
# [Internet->Proxy]
# ip rule:  from all to 10.10.1.0/24 iif enp4s0 lookup 100
# ip route: (100) default via 10.10.2.1 dev proxy-veth0
# [Proxy->Client]
# ip rule:  from all to 10.10.1.0/24 iif proxy-veth0 lookup 101
# ip route: (101) default via 10.10.1.1 dev client-veth0

Problem is, When I ping 8.8.8.8 from Client, within client netns, source ip masquerading does not happen. iptables masquerade rule does not match and defaults to ACCEPT . I expect that tcpdump on enp4s0 shows 172.16.202.30 --> 8.8.8.8, but it shows 10.10.1.1 --> 8.8.8.8, without source IP masquerading.

I recorded pcap on internet line to clarify SNAT does not occur. client_to_goolge is recorded from separate machine outside enp4s0:

$ tcpdump -r client_to_google -n
reading from file client_to_google, link-type EN10MB (Ethernet)
23:35:40.852257 IP 10.10.1.1 > 8.8.8.8: ICMP echo request, id 14867, seq 1, length 64
23:35:41.865269 IP 10.10.1.1 > 8.8.8.8: ICMP echo request, id 14867, seq 2, length 64

When I checked on iptables mangle table, packets flows by given policy:

  PREROUTING: client-veth0, 10.10.1.1 --> 8.8.8.8
  POSTROUTING: proxy-veth0, 10.10.1.1 --> 8.8.8.8
  PREROUTING: proxy-veth0, 10.10.1.1 --> 8.8.8.8
  POSTROUTING: enp4s0, 10.10.1.1 --> 8.8.8.8

However, when I change masquerade rule on proxy-veth0 out interface, like this iptables: -t nat -A POSTROUTING -s 10.10.10.1/32 -o proxy-veth0 -j MASQUERADE, masquerading happens. That is 10.10.2.2 --> 8.8.8.8 packets are captured.

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
...
11       0     0 MASQUERADE  all  --  *      enp4s0  10.10.1.1            0.0.0.0/0
12       1    84 MASQUERADE  all  --  *      proxy-veth0  10.10.1.1            0.0.0.0/0

Above table shows that rule #11 enp4s0 output condition did not trigger. Rule #12 was inserted after several test with rule #11. Rule #12 shows that proxy-veth0 output condition did trigger. Are there any differences between enp4s0 master nic and proxy-veth0 virthual interface with iptables?

Any comments will be appreciated deeply, Thank you.