MoWo's questions

I am troubleshooting an AWS RDS Postgres instance that has been restarted by AWS several times in the last few days, very likely due to resource constraints. It's a testing DB that usually doesn't do much but we recently put some higher load onto it. I found that the DB's EBS volume (200GB gp3) depleted its throughput credits and that the times of the DB restarts coincided pretty well with the EBSByteBalance% metric reaching zero. Then when the DB gets restarted, the volume apparently gets a fresh set of burst credits as can be seen in the screenshot below:

The credits now drop slightly slower as we have eased the load on the DB but they are still dropping. When I look at the current read and write throughput metrics, they seem to sum up to just about 5 to 7 MiB/s with occasional spikes:

Based on the information found here at Amazon RDS DB instance storage the baseline throughput for a gp3 volume below 400gb should be 125MiB/s. So can anyone help me explain why the EBSByteBalance% metric keeps decreasing in this scenario? Thanks!

I have an EC2 machine running Ubuntu 20.04 with 2 ethernet interfaces. They are both connected to the same subnet and both are reachable from the outside. I have created 2 custom route tables, one for each interface and both just containing the route entries for the machine itself, the immediate subnet and a default gateway via the corresponding interface. I can add a rule for all traffic to use either of the custom route tables and it works: the machine will use the specified interface and the corresponding ip address for outbound communication. However if I add rules to use either custom route table based on packet markings that I apply via iptables, everything stops working and no traffic is leaving my machine.

Here are the main bits of relevant configuration:

$ sudo ip route
10.0.11.0/24 dev ens6 proto kernel scope link src 10.0.11.52
10.0.11.0/24 dev ens5 proto kernel scope link src 10.0.11.201

$ sudo ip route show table if1
default via 10.0.11.1 dev ens5 proto static metric 100
10.0.11.0/24 dev ens5 proto static scope link src 10.0.11.52
10.0.11.201 dev ens5 proto static scope link metric 100

$ sudo ip route show table if2
default via 10.0.11.1 dev ens6 proto static metric 200
10.0.11.0/24 dev ens6 proto static scope link src 10.0.11.201
10.0.11.52 dev ens6 proto static scope link metric 200

$ ip rule
0:      from all lookup local
46:     from all fwmark 0x3 lookup if1
47:     from all fwmark 0x2 lookup if2
50:     from 10.0.11.201 lookup if1
100:    from 10.0.11.52 lookup if2
32766:  from all lookup main
32767:  from all lookup default

$ sudo iptables -S -t mangle
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A OUTPUT -m owner --uid-owner 1002 -j MARK --set-xmark 0x3/0xffffffff
-A OUTPUT -m owner --uid-owner 1003 -j MARK --set-xmark 0x2/0xffffffff

My goal is to route traffic from the user with uid 1002 over interface ens5 (if1) and traffic from user with uid 1003 via ens6 (if2) but I'm doing something wrong somewhere and it drives me nuts.... I can see that marked packets are being treated differently, so something is happening. If I assign a default route to the main routing table and then just mark traffic for uid 1002, all users can make outbound connections except for uid 1002. So I know the marking works, I know marked packets get routed differently and I know my custom route tables work but somewhere in this constellation there's an error and I can't find it. Any help would be greatly appreciated at this point!

--edit--

I arrived at this by basing my setupn on the approach outlined here: Trying to route outgoing http requests through VPN on Ubuntu Server 12.04 and some customizations as I don't wanna route traffic based on specific ports but originating uid.

I have an Ubuntu 20.04 machine with 2 ethernet interfaces with 2 IP addresses each. It's an AWS EC2 instance and each of the 4 IP addresses has an EIP attached to it via NAT. Both interfaces connect to the same internal subnet. The setup looks like this:

EC2 Machine:

- eni1:

• private-IP1 -> public-IP1
• private-IP2 -> public-IP2

- eni2:

• private-IP3 -> public-IP3
• private-IP4 -> public-IP4

All 4 addresses are reachabale from the outside so that seems to be all fine. However for outgoing traffic currently always private-IP1 (and thus public-IP1) gets used. I want to specify that individual SSH users use specific IP addresses, so they'll come from the corresponding public IP when talking to services on the internet i.e.

user1 -> private-IP1

user2 -> private-IP2

user3 -> private-IP3

user4 -> private-IP4

What's the best way of achieving this result?

We have a bunch of backend servers in the form of EC2 instances based in a private subnet in AWS VPC that need to talk to a 3rd party API. This API is limiting the requests we can send based on the originating ip address and while scaling our setup we have started hitting the limits on the IP of the NAT gateway that is used for all outbound traffic.

Thus I want to setup a proxy for outbound traffic with several EIPs attached. For testing I am currently using an Amazon Linux 2 instance with 2 ENIs with 2 EIPs attached each. The backend servers open an SSH tunnel to the outbound proxy and map the 3rd party API to a local port, an entry in the servers hosts file redirects all traffic to that hostname to localhost and this setup is working fine in general but outbound traffic from the proxy is always using only the first associated EIP.

So my setup looks like this:

ENI1: eth0
private IP1: 10.0.11.81
private IP2: 10.0.11.82

ENI2: eth1
private IP3: 10.0.11.52
private IP4: 10.0.11.53

original route table:
default via 10.0.11.1 dev eth0
default via 10.0.11.1 dev eth1 metric 10001
10.0.11.0/24 dev eth0 proto kernel scope link src 10.0.11.81
10.0.11.0/24 dev eth1 proto kernel scope link src 10.0.11.52
169.254.169.254 dev eth0

I now want to be able to specify which backend server uses which EIP when calling the API via the outbound proxy. My first try was the following:

• setup 4 different users on the proxy host
• add iptable rules for each user like so: iptables -t nat -m owner --uid-owner user1 -A POSTROUTING -j SNAT --to IP1 etc.
• this works for the 2 IPs that are attached to the primary ENI (ie eth0 in the machine) but does not work for the 2 IPs associated with the second ENI (eth1)
• adding -o eth1 to the statement did not work either

My next try was to create custom routing tables for each IP address and matching them to policy rules:

• create custom route table i.e. for IP3:

default via 10.0.11.1 dev eth1
10.0.11.0/24 dev eth1 proto static scope link src 10.0.11.52
169.254.169.254 dev eth1 scope link

• create iptables rule to mark traffic originating from user3: -A OUTPUT -m owner --uid-owner 1003 -j MARK --set-xmark 0x3/0xffffffff
• create rule to utilize custom route table for all packets marked 3: 32763: from all fwmark 0x3 lookup ip3
• this again does not work. packets do get treated differently. all users can communicate with the world except for user3 in the above example.

What am I doing wrong? Is there something trivial I am missing or is my entire approach doomed to fail? I'm very open to suggestions, both on getting this setup working as well as alternative approaches...

Thanks a lot in advance!