I am currently in the process of setting secure as possible KVM based server. The VM host should be more or less invisible to the internet and only VM guests running single services should be exposed. VM guests should have no internet access (--> no NAT masquerade).
Now I would like to expose HTTP(s) ports of the VM guest 192.168.122.183 to the internet. All existing documentation uses iptables which is almost deprecated. So I prefer a solution using nftables.
According to the documentation, this should be fairly simple, so my nft list ruleset looks accordingly:
table ip nat {
chain prerouting {
type nat hook prerouting priority 0; policy accept;
iif "enp35s0" tcp dport { http, https } dnat to 192.168.122.183
}
}
table inet filter {
chain input {
type filter hook input priority 0; policy accept;
}
chain forward {
type filter hook forward priority 0; policy accept;
}
chain output {
type filter hook output priority 0; policy accept;
}
}
Also ip forwarding is activated sysctl -a | grep forward:
net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.enp35s0.bc_forwarding = 0
net.ipv4.conf.enp35s0.forwarding = 1
net.ipv4.conf.enp35s0.mc_forwarding = 0
net.ipv4.conf.virbr10.bc_forwarding = 0
net.ipv4.conf.virbr10.forwarding = 1
net.ipv4.conf.virbr10.mc_forwarding = 0
net.ipv4.conf.virbr10-dummy.bc_forwarding = 0
net.ipv4.conf.virbr10-dummy.forwarding = 1
net.ipv4.conf.virbr10-dummy.mc_forwarding = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
Accessing the web server from the VM host works perfectly but from the internet I don't get response. What am I missing?
Might be related to this question.
Here is an exerpt from tcpdump from the host. As seen request from the VM host is answered correctly and also the external request is popping up but is never answered. Any ideas why?
23:43:28.048714 IP 192.168.122.1.35584 > 192.168.122.183.http: Flags [S], seq 502099991, win 64240, options [mss 1460,sackOK,TS val 3361102460 ecr 0,nop,wscale 7], length 0
23:43:28.048746 IP 192.168.122.183.http > 192.168.122.1.35584: Flags [S.], seq 3142491522, ack 502099992, win 65160, options [mss 1460,sackOK,TS val 519662698 ecr 3361102460,nop,wscale 7], length 0
23:43:28.048801 IP 192.168.122.1.35584 > 192.168.122.183.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 3361102460 ecr 519662698], length 0
23:43:28.048846 IP 192.168.122.1.35584 > 192.168.122.183.http: Flags [P.], seq 1:143, ack 1, win 502, options [nop,nop,TS val 3361102460 ecr 519662698], length 142: HTTP: GET / HTTP/1.1
23:43:28.048851 IP 192.168.122.183.http > 192.168.122.1.35584: Flags [.], ack 143, win 508, options [nop,nop,TS val 519662698 ecr 3361102460], length 0
23:43:28.048960 IP 192.168.122.183.http > 192.168.122.1.35584: Flags [P.], seq 1:325, ack 143, win 508, options [nop,nop,TS val 519662698 ecr 3361102460], length 324: HTTP: HTTP/1.1 200 OK
23:43:28.049015 IP 192.168.122.1.35584 > 192.168.122.183.http: Flags [.], ack 325, win 501, options [nop,nop,TS val 3361102460 ecr 519662698], length 0
23:43:45.263545 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:a7:a9.8002, length 35
23:43:45.537492 IP external.dip0.t-ipconnect.de.46866 > 192.168.122.183.http: Flags [S], seq 1122615229, win 64240, options [mss 1452,sackOK,TS val 2536247181 ecr 0,nop,wscale 7], length 0
23:43:47.247528 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:a7:a9.8002, length 35
23:43:47.553490 IP external.dip0.t-ipconnect.de.46866 > 192.168.122.183.http: Flags [S], seq 1122615229, win 64240, options [mss 1452,sackOK,TS val 2536249197 ecr 0,nop,wscale 7], length 0
For reference my ip addr output:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp35s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether a8:a1:59:0e:**:** brd ff:ff:ff:ff:ff:ff
inet 95.217.***.***/26 brd 95.217.120.127 scope global enp35s0
valid_lft forever preferred_lft forever
inet6 2a01:4f9:**:****::2/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::aaa1:59ff:****:*****/64 scope link
valid_lft forever preferred_lft forever
3: virbr10-dummy: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr10 state UNKNOWN group default qlen 1000
link/ether 52:54:00:de:a7:a9 brd ff:ff:ff:ff:ff:ff
4: virbr10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:de:a7:a9 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr10
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fede:a7a9/64 scope link
valid_lft forever preferred_lft forever
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr10 state UNKNOWN group default qlen 1000
link/ether fe:54:00:94:b9:ce brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe94:b9ce/64 scope link
valid_lft forever preferred_lft forever
Another question answered that virt-manager is incompatible with nftables as it still uses iptables. But the iptable rules are empty and the iptables modules (especially iptable_nat) are not loaded
# lsmod | grep ip
nft_chain_route_ipv4 16384 0
nft_chain_nat_ipv4 16384 1
ipt_MASQUERADE 16384 0
nf_nat_ipv4 16384 2 ipt_MASQUERADE,nft_chain_nat_ipv4
nf_nat 36864 2 nft_nat,nf_nat_ipv4
ipt_REJECT 16384 0
nf_reject_ipv4 16384 1 ipt_REJECT
nf_conntrack 172032 5 xt_conntrack,nf_nat,nft_nat,ipt_MASQUERADE,nf_nat_ipv4
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
nf_tables 143360 13 nft_chain_route_ipv4,nft_compat,nft_nat,nft_chain_nat_ipv4,nft_counter,nf_tables_set
ip_tables 28672 0
x_tables 45056 7 xt_conntrack,nft_compat,xt_tcpudp,ipt_MASQUERADE,xt_CHECKSUM,ipt_REJECT,ip_tables