Ondra Sniper Flidr's questions

I'm playing with vxlans on ubuntu 18.04 right now and I just found something strange. I have 2 servers with 4 containers on each. Containers are connected to bridges br1 - br4. There are vx1 - vx4 interfaces connected to corresponding bridges. Whole setup looks like this:

             node1                               node2
------- ------- ------- -------     ------- ------- ------- -------     
| c1a | | c2a | | c3a | | c4a |     | c1b | | c2b | | c3b | | c4b |
------- ------- ------- -------     ------- ------- ------- -------
   |       |       |       |           |       |       |       |
------- ------- ------- -------     ------- ------- ------- -------     
| br1 | | br2 | | br3 | | br4 |     | br1 | | br2 | | br3 | | br4 |
------- ------- ------- -------     ------- ------- ------- -------
   |       |       |       |           |       |       |       |
------- ------- ------- -------     ------- ------- ------- -------     
| vx1 | | vx2 | | vx3 | | vx4 |     | vx1 | | vx2 | | vx3 | | vx4 |
------- ------- ------- -------     ------- ------- ------- -------
   |       |       |       |           |       |       |       |
-------------------------------     -------------------------------     
|            eth0             |     |            eth0             |
-------------------------------     -------------------------------
               |                                   |
               \-----------------------------------/

Interfaces on nodes itself are same (at least configuration of bridges and vxlans) and looks like this:

auto vx1
iface vx1 inet manual
    mtu 1450
    pre-up ip link add vx1 type vxlan id 2584 group 239.0.3.20 dstport 8472 port 32768 61000 dev eth0 || true
    up ip link set vx1 up
    down ip link set vx1 down
    post-down ip link del vx1 || true

auto vx2
iface vx2 inet manual
    mtu 1450
    pre-up ip link add vx2 type vxlan id 1428 group 239.0.3.20 dstport 8472 port 32768 61000 dev eth0 || true
    up ip link set vx2 up
    down ip link set vx2 down
    post-down ip link del vx2 || true

auto vx3
iface vx3 inet manual
    mtu 1450
    pre-up ip link add vx3 type vxlan id 2584 group 239.0.3.50 dstport 8472 port 32768 61000 dev eth0 || true
    up ip link set vx3 up
    down ip link set vx3 down
    post-down ip link del vx3 || true

auto vx4
iface vx4 inet manual
    mtu 1450
    pre-up ip link add vx4 type vxlan id 58996 group 239.0.3.14 dstport 8472 port 32768 61000 dev eth0 || true
    up ip link set vx4 up
    down ip link set vx4 down
    post-down ip link del vx4 || true

auto br1
iface br1 inet manual
    bridge_ports vx1
    bridge_stp off
    bridge_fd 0
    bridge_hello 2
    bridge_maxage 12

auto br2
iface br2 inet manual
    bridge_ports vx2
    bridge_stp off
    bridge_fd 0
    bridge_hello 2
    bridge_maxage 12

auto br3
iface br3 inet manual
    bridge_ports vx3
    bridge_stp off
    bridge_fd 0
    bridge_hello 2
    bridge_maxage 12

auto br4
iface br4 inet manual
    bridge_ports vx4
    bridge_stp off
    bridge_fd 0
    bridge_hello 2
    bridge_maxage 12

This setup AFAIK should be working (all vxlan have different pair of group IP and vxlan id) but system cannot setup vx3 because of duplicity of vxlan id (regardless there is different group ip used). Is there any fix for this or there is strict limit vxlan id must be unique even across multiple group ips?

I have to solve this problem. I have two memcache servers (SM1 and SM2) and HAProxy loadbalancer in front of them. HAProxy is configured in active/passive mode, when SM2 has backup command. All traffic is routed to only one server at any time, switching only when active server fails.

I need to ensure memcache is clean before server goes UP. Our application cannot detect if data in memcache is too old and will use it (it can lead to some problems). How can I send flush_all to server before is marked as UP? Is it even possible? Or is better to switch all application to redis (which we're using to store sessions data) with replication? Thanks a lot.

I would like to use Nat Gateway (not Nat Instance on EC2!) from another VPC to route my traffic from peered VPCs to the Internet. My infrastructure looks like this:

/---------------------VPC-LIVECHAT---------------------\
| /---Subnet A---\  /---Subnet B---\  /---Subnet C---\ |
| |              |  |              |  |              | |
| \-10.10.0.0/24-/  \-10.10.1.0/24-/  \-10.10.2.0/24-/ |
\------------------------------------------------------/
               |                        |
               | VPC Peering Connection |
               |                        |
/----------------------VPC-COMMON----------------------\
| /---Subnet A---\  /---Subnet B---\  /---Subnet C---\ |
| |  /--------\  |  |  /--------\  |  |  /--------\  | |
| |  | NAT GW |  |  |  | NAT GW |  |  |  | NAT GW |  | |
| |  \--------/  |  |  \--------/  |  |  \--------/  | |
| \-10.10.3.0/24-/  \-10.10.4.0/24-/  \-10.10.5.0/24-/ |
\------------------------------------------------------/
               |                        |
               | VPC Peering Connection |
               |                        |
/---------------------VPC-DATABASE---------------------\
| /---Subnet A---\  /---Subnet B---\  /---Subnet C---\ |
| |              |  |              |  |              | |
| \-10.10.6.0/24-/  \-10.10.7.0/24-/  \-10.10.8.0/24-/ |
\------------------------------------------------------/

My idea:

• I will setup VPC-COMMON with subnets and Nat Gateway in each subnet (one subnet per AZ)
• I will setup VPC-LIVECHAT and VPC-DATABASE VPCs, create VPC Peering connections
• In VPC-COMMON subnets there will be route 0.0.0.0/0 -> Nat Gateway in same subnet
• In VPC-LIVECHAT and VPC-DATABASE subnets (all of them) there will be route VPC-COMMON CIDR -> VPC Peering Connection
• In each subnet of VPC-LIVECHAT and VPC-DATABASE there will be route 0.0.0.0/ -> Nat gateway in corresponding subnet of VPC-COMMON (subnets A will use NAT GW in VPC-COMMON subnet A and so on...)

I think this setup should work pretty well, it is just routed VLANs. But not in AWS. AWS don't want to allow me use Nat Gateway in different VPC in route table with error

"route table rtb-293fa54d and interface interface-c2002e9e belong to different networks"

I cannot use Private IP of Nat Gateway in AWS too, AWS doesn't suport IP addresses in route target (I would really love to know why).

I'm using CloudFormation and my route definition looks like this:

"RoutePrivate3ToNatInCommon" : {
    "Type" : "AWS::EC2::Route",
    "Condition" : "IsNotVpcCommon",
    "Properties" : {
        "DestinationCidrBlock" : "0.0.0.0/0",
        "RouteTableId" : { "Ref" : "PrivateSubnet3RoutingTable" },
        "NatGatewayId" : { "Fn::GetAtt" : [ "NatGatewaySettingsForNotCommon", "NatGatewayAZC" ] }
    }
}

NatGatewaySettingForNotCommon is my Custom lambda-backed resource which helps me to get list of Nat Gateways per availability zone.

Is there any way how to achieve this setup? I will have about 10 VPCs per region, 3 private subnets in each of them and I really don't want to setup (and pay for) 30 Nat Gateways. This looks like regular "non-cloudy" network setup so there should be no problem to realize it in cloud. Or is it?

I'm facing strange problem with iptables right now. Simply, I want to redirect traffic from port 514 to port 5140. I'm using this IPTables command to achieve it:

iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to-port 5140

But I still get error about no chain

iptables: No chain/target/match by that name.

There is definitelly PREROUTING chain in nat table, this is iptables -t nat -nL

root@VPS-LOGGER-TMP:~# iptables -t nat -nL
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination    

I'm using Debian Wheezy and I have loaded all needed modules:

root@VPS-LOGGER-TMP:~# lsmod |grep nat
iptable_nat            12928  0 
nf_nat                 18242  1 iptable_nat
nf_conntrack_ipv4      14078  4 nf_nat,iptable_nat
nf_conntrack           52720  4 xt_state,nf_conntrack_ipv4,nf_nat,iptable_nat
ip_tables              22042  2 iptable_filter,iptable_nat
x_tables               19118  9 ip_tables,iptable_filter,xt_multiport,ip6_tables,ip6table_filter,xt_state,xt_tcpudp,ipt_REJECT,iptable_nat

Can you tell me what I'm doing wrong? As far as I know this should work perfectly. Thanks a lot.

I have weird problem with openvpn setup between AWS and GoGrid datacenters. My network looks like this

/----------------\               /----------------\               /------------------\               /----------------\
|  VPS-DEVEL.gg  |               |    VPS-VPN.gg  |               |   VPS-VPN.aws    |               | VPS-PROVIS.aws |
| 10.160.64.7/24 | eth1 --- eth1 | 10.160.64.9/24 | tun0 --- tun0 | 10.160.48.219/24 | eth0 --- eth0 | 10.160.52.8/24 |
\----------------/               \----------------/               \------------------/               \----------------/

I can ping from aws to gogrid without problem (both VPS-DEVEL.gg and VPS-VPN.gg from both aws VMs), but I cannot ping from gogrid to AWS.

My routing table on VPS-VPN.gg looks this:

[root@VPSVPN ~]# route -n
Směrovací tabulka v jádru pro IP
Adresát         Brána           Maska           Přízn Metrik Odkaz  Užt Rozhraní
169.254.4.1     164.40.132.83   255.255.255.255 UGH   0      0        0 eth0
169.254.4.2     10.160.64.9     255.255.255.255 UGH   0      0        0 eth1
10.8.0.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
164.40.132.80   0.0.0.0         255.255.255.240 U     0      0        0 eth0
10.8.0.0        10.8.0.2        255.255.255.240 UG    0      0        0 tun0
10.159.254.0    10.160.64.1     255.255.255.0   UG    0      0        0 eth1
10.160.64.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.160.0.0      10.8.0.2        255.255.192.0   UG    0      0        0 tun0
0.0.0.0         164.40.132.81   0.0.0.0         UG    0      0        0 eth0

My routing table on VPS-VPN.aws:

admin@ip-10-160-48-219:~$ sudo route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.160.48.1     0.0.0.0         UG    0      0        0 eth0
10.8.0.0        10.8.0.9        255.255.255.240 UG    0      0        0 tun0
10.8.0.9        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.160.48.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.160.64.0     10.8.0.9        255.255.255.0   UG    0      0        0 tun0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0

My openvpn server config (gogrid side):

[root@VPSVPN ~]# cat /etc/openvpn/server.conf
port 1194
proto udp
dev tun
ca ca.crt
cert vpsvpn.crt
key vpsvpn.key  # This file should be kept secret
dh dh1024.pem
server 10.8.0.0 255.255.255.240
ifconfig-pool-persist ipp.txt
client-config-dir /etc/openvpn/ccd # ghor
client-to-client # ghor
keepalive 10 120
comp-lzo
user nobody
group nobody
persist-key
persist-tun
max-clients 100
status /var/log/openvpn-status.log
log-append  /var/log/openvpn.log
verb 11
route 10.160.0.0 255.255.192.0
push "route 10.160.64.0 255.255.255.0"

My openvpn client config (aws side):

admin@ip-10-160-48-219:~$ cat /etc/openvpn/gogrid/gogrid.ovpn 
client
dev tun
proto udp
remote 164.40.132.83 1194
resolv-retry infinite
nobind
persist-key
persist-tun
comp-lzo
verb 3
ca /etc/openvpn/gogrid/ca.crt
cert /etc/openvpn/gogrid/test-eu-west-1-aws.crt
key /etc/openvpn/gogrid/test-eu-west-1-aws.key
askpass /etc/openvpn/gogrid/test-eu-west-1-aws.pass

TCPDump shows this:

1. Ping from aws (both VMs) goes on VPS-VPN.aws tun0 interface into vpn tunnel, comes to tun0 on VPS-VPN.gg and reply goes back right
2. Ping from gogrid goes on VPS-VPN.gg tun0 interface into vpn tunnel but didn't arrive to tun0 interface on VPS-VPN.aws
3. Ping from VPS-VPN.gg to VPS-VPN.aws IP of tun0 (10.8.0.10) works well

Both VPS-VPN has enabled ip_forward.

IPTables on VPS-VPN.aws looks this, AWS security groups is set to allow all traffic from everywhere (I don't like to use SecGroups when I can use iptables on VMs):

admin@ip-10-160-48-219:~$ sudo iptables -nvL
Chain INPUT (policy ACCEPT 867 packets, 68073 bytes)
 pkts bytes target        prot opt in     out     source            destination         
 1426  117K fail2ban-ssh  tcp  --  *      *       0.0.0.0/0         0.0.0.0/0            multiport dports 22

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1360  105K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 743 packets, 72322 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain fail2ban-ssh (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1400  115K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0          

admin@ip-10-160-48-219:~$ sudo iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 1743 packets, 105K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 1359 packets, 69760 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 390 packets, 40130 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target      prot opt in     out     source              destination         
  748 67734 MASQUERADE  all  --  *      eth0    0.0.0.0/0           0.0.0.0/0           
   17  1428 MASQUERADE  all  --  *      tun0    0.0.0.0/0           0.0.0.0/0           

I'm using 10.160.0.0/18 in routing just because I will have more VPCs with subnets in this range. Each AWS VPC has subnet /21. Everything is in subnet 10.160.64.0/24 in GoGrid side and aws routing table is set to route everything for this subnet to VPS-VPN.aws instance. This is working, I can ping GoGrid from AWS.

Can you point me where I'm making some mistake? This setup AFAIK should be working for both directions. Thanks a lot.