Doug Smythies's questions

Summary: In a bridged KVM/QEMU configuration, network packets destined to guest VM do not get there.

Configuration: The host is an up to date Ubuntu 20.04.2 LTS server; The guest is any of 3 VMs, a very old 16.04 Ubuntu server, an old Ubuntu 20.04 desktop, and a brand new Ubuntu 21.04 desktop. The first 2 VMs are being converted from non-bridged, NAT'd, and the 3rd was created specifying bridged networking. Ultimately the VMs will get their IP addresses from the main LAN, via DHCP, but for now and for better debugging information, they are using static IP addresses.

The host bridge definition, /etc/netplan/01-netcfg.yaml, (this is one of many many attempts):

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0:
      dhcp4: no
  bridges:
    br0:
#      interfaces: [ enp3s0 ]
      dhcp4: yes
#      dhcp6: no
#      link-local: [ ]
      interfaces:
        - enp3s0
#      parameters:
#        stp: true
#        forward-delay: 4

The virtual stuff, /etc/libvirt/qemu/networks/br0.xml:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh net-edit br0
or other application using the libvirt API.
-->

<network>
  <name>br0</name>
  <uuid>40a8752c-d074-4802-bae8-b0aef95d9c99</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

Notes: many versions of bridge .xml files have been tried, including different names, and different references use differing techniques. The Ubuntu Serverguide references this which says the name and bridge name have to be the same, but other references do not have them the same. Once the bare file has been created with nano, these commands:

virsh net-define br0.xml
virsh net-autostart br0
virsh net-start br0

were used to add and configure it. The default, NAT'd way was unlinked, from the autostart directory, so as not to. Eventually, it was undefined. The result:

$ virsh net-list --all
 Name      State      Autostart   Persistent
----------------------------------------------
 br0       active     yes         yes

At this point, there are no iptables rules at all after re-boot. However the VMs do not have network access. Note that some references mention special iptables rules and special attributes for the br_netfilter module, all of which have been tried. This question is already long enough without going into all the detail of variations tried here.

Debug Details: No matter the configuration variant, the base issue is always the same, VM destined packets do not seem to arrive at the host, at least as seen by tcpdump. However, broadcast type packets do arrive and do get to the client VM.

This example will use 192.168.111.59 (MAC: 52:54:00:60:ea:0e), the 16.04 server VM, and 192.168.111.132, a raspberry-pi, on the LAN. The 20.04 host server is at 192.168.111.136. The netmask is 24 bits, 255.255.255.0. The gateway and DHCP server is a Debian server (on which bridged guest VMs work fine, by the way).

First tpcudmp as seen from the raspberry-pi during ping:

doug@rpi2:~ $ sudo tcpdump -n -tttt -i eth0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-23 08:33:19.363553 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 08:33:19.487239 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 14, length 64
2021-04-23 08:33:20.363542 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 08:33:20.527250 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 15, length 64
2021-04-23 08:33:21.567215 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 16, length 64
2021-04-23 08:33:22.607228 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 17, length 64
2021-04-23 08:33:23.372351 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 08:33:23.647228 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 18, length 64
2021-04-23 08:33:24.371431 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 46

Observe the VM is sending out packets just fine, as seen via all the ARP activity. However it never replies to anything. Now lets watch the same activity from the host, noting tcpdump output is the same for any of interfaces br0 or enp3s0 or vnet0.

$ sudo tcpdump -n -tttt -i br0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-23 08:40:38.837608 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:39.837159 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:40.837122 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:43.842985 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:44.840895 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:45.840991 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:48.848508 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:49.848895 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:50.848871 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:51.514011 ARP, Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 08:40:52.928400 ARP, Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 08:40:53.853881 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:54.852472 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28

Observe every so often the VM does respond, but later on we'll see that it is to a broadcast packet. It also seems as though a problem is that 192.168.111.1 isn't responding. It is, and for whatever reason, that packets are not seen here at the tcpdump level. Also notice no ICMP packets from the raspberry-pi. Now, show the gateway responding (this is "br0" on a different computer. EDIT: replaced with a better capture example, hence the different timestamps):

$ sudo tcpdump -n -tttt -e -i br0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
2021-04-23 22:25:17.434415 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:17.434432 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:20.440843 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:20.440859 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:21.438316 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:21.438332 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:22.438266 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:22.438283 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:25.446312 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:25.446329 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:26.446195 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:26.446211 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28

Observe the VM outgoing packets, and ARP does complete. I can't figure out how to copy and paste from the VM, which I communicate with via VNC, but it shows some completed, but stale ARP entries in response to the ip neigh command and tcpdump shows some ARP and broadcast packets from the LAN.

Other information (MACs not relevant to this question have been hidden):

$ brctl show br0
bridge name     bridge id               STP enabled     interfaces
br0             8000.3c7c3f0d9983       no              enp3s0
                                                        vnet0
$ brctl showmacs br0
port no mac addr                is local?       ageing timer
  1     xx:xx:xx:xx:xx:xx       no                 0.00
  1     3c:7c:3f:0d:99:83       yes                0.00
  1     3c:7c:3f:0d:99:83       yes                0.00
  2     52:54:00:60:ea:0e       no                 1.68
  1     xx:xx:xx:xx:xx:xx       no                 2.14
  1     xx:xx:xx:xx:xx:xx       no                36.84
  1     xx:xx:xx:xx:xx:xx       no                89.57
  1     xx:xx:xx:xx:xx:xx       no               226.51
  1     xx:xx:xx:xx:xx:xx       no                13.28
  1     xx:xx:xx:xx:xx:xx       no               165.68
  1     xx:xx:xx:xx:xx:xx       no               165.68
  1     xx:xx:xx:xx:xx:xx       no               265.02
  1     xx:xx:xx:xx:xx:xx       no                27.62
  2     fe:54:00:60:ea:0e       yes                0.00
  2     fe:54:00:60:ea:0e       yes                0.00

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 3c:7c:3f:0d:99:83 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:7c:3f:0d:99:83 brd ff:ff:ff:ff:ff:ff
    inet 192.168.111.136/24 brd 192.168.111.255 scope global dynamic br0
       valid_lft 51547sec preferred_lft 51547sec
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:60:ea:0e brd ff:ff:ff:ff:ff:ff

EDIT: interestingly, all ARP packets from my D-Link AC2600 Wi-Fi Gigabit Router, configured as a switch, always appear on the host and get to the VM and are replied to:

$ sudo tcpdump -n -tttt -e -i br0 ether host aa:aa:aa:aa:aa:aa
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-23 22:45:51.463524 aa:aa:aa:aa:aa:aa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.59 (ff:ff:ff:ff:ff:ff) tell 192.168.111.58, length 46
2021-04-23 22:45:51.463631 52:54:00:60:ea:0e > aa:aa:aa:aa:aa:aa, ethertype ARP (0x0806), length 42: Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 22:46:51.466955 aa:aa:aa:aa:aa:aa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.59 (ff:ff:ff:ff:ff:ff) tell 192.168.111.58, length 46
2021-04-23 22:46:51.467030 52:54:00:60:ea:0e > aa:aa:aa:aa:aa:aa, ethertype ARP (0x0806), length 42: Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 22:47:51.466889 aa:aa:aa:aa:aa:aa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.59 (ff:ff:ff:ff:ff:ff) tell 192.168.111.58, length 46
2021-04-23 22:47:51.466965 52:54:00:60:ea:0e > aa:aa:aa:aa:aa:aa, ethertype ARP (0x0806), length 42: Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 22:48:51.479096 aa:aa:aa:aa:aa:aa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.59 (ff:ff:ff:ff:ff:ff) tell 192.168.111.58, length 46
2021-04-23 22:48:51.479178 52:54:00:60:ea:0e > aa:aa:aa:aa:aa:aa, ethertype ARP (0x0806), length 42: Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28

EDIT 3 - New configuration test: In an attempt to reduce the number of variables the following was done:

• An about to be retired Ubuntu 16.04 server was powered up, providing a new isolated LAN.
• the host Ubuntu 20.04 server was connected directly to the LAN side NIC of the 16.04 server. No switches were involved at all, just one long Ethernet cable.
• Things were tested, and seemed to work fine. Access to everything was via ssh from my main LAN out through my main static WAN IP and back in via my test WAN static IP to the old 16.04 server. Then a chained ssh session from there to the 20.04 host server.
• The Ubuntu 16.04 VM client was started on the host.
• "ping" was attempted from the old 16.04 gateway server to the client.
• The results were the same as the original configuration.

tcpdump on gateway old 16.04 server:

doug@DOUG-64:~$ sudo tcpdump -n -tttt -e -i enp2s0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-26 15:10:00.701941 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:00.701965 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:01.699156 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:01.699169 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:02.699141 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:02.699154 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:05.707404 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:05.707417 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:06.707097 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:06.707110 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:07.707094 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:07.707107 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28

tcpdump on 20.04 host server:

doug@s19:~$ sudo tcpdump -n -tttt -e -i br0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-26 15:11:35.801771 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:36.800497 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:37.800491 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:40.807062 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:41.804469 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:42.804444 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:45.812553 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:46.812405 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:47.812398 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28

As a side note: From my chained ssh session to the 20.04 host server, I can chain again and ssh to the VM client just fine.

Conclusion: Something is wrong right at the the Link Layer on the Ubuntu 20.04 server such that the incoming packets are not even "seen" by tcpdump, nor get to the VM guest. Diagram.

EDIT 4: Comparing to information provided by Christian Ehrhardt a potential difference is on my system is the br0 MAC list might be incorrect, with the fist byte replaced. Note: non relevant MAC deleted, 3 VMs running:

doug@s19:~$ brctl showmacs br0
port no mac addr                is local?       ageing timer
  1     3c:7c:3f:0d:99:83       yes                0.00  <<< enp3s0, br0
  1     3c:7c:3f:0d:99:83       yes                0.00  <<< enp3s0, br0
  4     52:54:00:22:2f:dc       no                 5.15  <<< VM 3
  2     52:54:00:60:ea:0e       no                 3.29  <<< VM 1
  3     52:54:00:60:ea:3e       no                12.67  <<< VM 2
  4     fe:54:00:22:2f:dc       yes                0.00  <<< vnet2
  4     fe:54:00:22:2f:dc       yes                0.00  <<< vnet2
  2     fe:54:00:60:ea:0e       yes                0.00  <<< vnet0
  2     fe:54:00:60:ea:0e       yes                0.00  <<< vnet0
  3     fe:54:00:60:ea:3e       yes                0.00  <<< vnet1
  3     fe:54:00:60:ea:3e       yes                0.00  <<< vnet1

For whatever reason Christian does not show a vnet, or whatever it is called on his system, connection to the bridge (I do not know it it is relevant or not):

$ brctl showmacs br0
port no mac addr        is local?   ageing timer
  2 52:54:00:48:40:69   no         2.36   <- Guest
  1 52:54:00:95:e4:2a   no         0.00   <- outside system
  1 52:54:00:9b:9b:0e   yes        0.00   <- Host
  1 52:54:00:9b:9b:0e   yes        0.00   <- Host

EDIT 5: Similar data to EDIT 4, but from Debian Server with 2 VMs running, which is working properly:

doug@s15:~$ sudo brctl showmacs br0
port no mac addr                is local?       ageing timer
  1     52:54:00:22:2f:dc       no                17.85
  2     52:54:00:27:1b:5e       no                18.48  <<< VM 1
  3     52:54:00:27:1b:ae       no                 2.14  <<< VM 2
  1     f4:8c:eb:c8:08:a0       no                18.48
  2     fe:71:fa:75:16:93       yes                0.00  <<< tap0 (VM1)
  2     fe:71:fa:75:16:93       yes                0.00  <<< tap0
  3     fe:e1:c5:2a:c7:e3       yes                0.00  <<< tap1 (VM2)
  3     fe:e1:c5:2a:c7:e3       yes                0.00  <<< tap1

EDIT 6: information from networkctl, the thing to note is that the Debian server shows "Master: br0" whereas the Ubuntu server doesn't:

Ubuntu:

doug@s19:~$ networkctl
IDX LINK   TYPE     OPERATIONAL SETUP
  1 lo     loopback carrier     unmanaged
  2 enp3s0 ether    enslaved    configured
  3 br0    bridge   routable    configured
  4 vnet0  ether    carrier     unmanaged

4 links listed.
doug@s19:~$ man networkctl
doug@s19:~$ networkctl^Cnetworkctl
doug@s19:~$ networkctl status vnet0
● 4: vnet0
             Link File: /usr/lib/systemd/network/99-default.link
          Network File: n/a
                  Type: ether
                 State: carrier (unmanaged)
                Driver: tun
            HW Address: fe:54:00:60:ea:0e
                   MTU: 1500 (min: 68, max: 65521)
  Queue Length (Tx/Rx): 1/1
      Auto negotiation: no
                 Speed: 10Mbps
                Duplex: full
                  Port: tp

Apr 30 07:40:51 s19 systemd-networkd[530]: vnet0: Link UP
Apr 30 07:40:51 s19 systemd-networkd[530]: vnet0: Gained carrier

Debian:

doug@s15:~$ networkctl
IDX LINK   TYPE     OPERATIONAL SETUP
  1 lo     loopback carrier     unmanaged
  2 enp3s0 ether    enslaved    configured
  3 enp1s0 ether    routable    configured
  4 br0    bridge   routable    configured
 10 tap0   ether    carrier     unmanaged

5 links listed.
doug@s15:~$ networkctl status tap0
● 10: tap0
             Link File: /usr/lib/systemd/network/99-default.link
          Network File: n/a
                  Type: ether
                 State: carrier (unmanaged)
                Driver: tun
            HW Address: fe:8a:6a:ce:18:9c
                   MTU: 1500 (min: 68, max: 65521)
                 QDisc: pfifo_fast
                Master: br0   <<<<< Different than Ubuntu
  Queue Length (Tx/Rx): 1/1
      Auto negotiation: no
                 Speed: 10Mbps
                Duplex: full
                  Port: tp

Question: What is wrong and how can I get bridged VMs working properly?

For 20.04 the Ubuntu serverguide has moved to a new location. Redirects have been added to the .htaccess file. However, they are not working for all URLs.

Note that the related parts of help.ubuntu.com are published from the master launchpad branch only once per day. The objective is to get this right, as I obviously didn't yesterday.

Working example:

https://help.ubuntu.com/20.04/serverguide/remote-administration.html.fr

goes to:

https://ubuntu.com/server/docs

Not working example:

https://help.ubuntu.com/20.04/serverguide/remote-administration.html

returns 404 Not Found.

The relevant portion of the .htaccess file:

# For 20.04, and likely onwards, the serveguide has moved.
# Don't try to be too clever, just force the base page and drop the rest.
#
RedirectMatch permanent ^/(stable|lts|20\.04)/(serverguide/.+\.html)\..* https://ubuntu.com/server/docs
RedirectMatch permanent ^/(stable|lts|20\.04)/(serverguide/.+\.pdf)\..* https://assets.ubuntu.com/ubuntu-server-guide

It seems to need the language extension to work. Will this work?

RedirectMatch permanent ^/(stable|lts|20\.04)/(serverguide/.+\.html) https://ubuntu.com/server/docs
RedirectMatch permanent ^/(stable|lts|20\.04)/(serverguide/.+\.pdf) https://assets.ubuntu.com/ubuntu-server-guide

I do not have a way to test this on my test server, and would like to get it right in the next publication.

EDIT: By deleting the "start of line" part of the expression, it turns out that I can debug this on my test server, where the location is several sub-directories in. With Muru's testing with the answer and my own testing, there is confidence in the solution.

References:
https://help.ubuntu.com/
https://code.launchpad.net/~ubuntu-core-doc/help.ubuntu.com/help.ubuntu.com
https://bazaar.launchpad.net/~ubuntu-core-doc/help.ubuntu.com/help.ubuntu.com/view/head:/.htaccess
Apache web server - how to strip language extentions

This question pertains to publishing the Ubuntu Serverguide on help.ubuntu.com. For the 20.04 LTS cycle, there will (O.K. might) be significant changes to the source code workflow for the Ubuntu Serverguide. Currently, translations are not being considered for this migration. Our best feedback has been that server admin types prefer English, even if it is not their first language. Before the decision becomes non reversible, we want to test it by doing the next point release of the 18.04 Ubuntu Serverguide in U.S. English only. Everything is ready, except for one issue:

We know for certain that many links, bookmarks, etc. exist with the language extension. Example:

https://help.ubuntu.com/lts/serverguide/networking.html.en-CA

And we want to have that scenario return this page instead:

https://help.ubuntu.com/lts/serverguide/networking.html

because the language specific versions will no longer exist, but returning a 404 Not Found error is undesirable.

The current version of an .htaccess file, with commented out previous attempts, is:

# unable to make below method work.
#RedirectMatch permanent ^(*\.html)\.*$ $1
#
# enable rewriting
RewriteEngine on
#RewriteRule ^(*\.html)\.*$ $1 [R=301, L]
#RewriteRule ^(*\.html)\.*$ $1
RewriteRule ^(*\.html)\.*$ $1 [PT]
#RewriteRule ^(*.html).*$ $1

Resulting in:

500 Internal Server Error

to the client, and this in the test server logs:

[Thu Jun 20 11:57:07.647838 2019] [core:alert] [pid 16079] [client 192.168.111.101:62992] /home/doug/public_html/linux/ubuntu-docs/help.ubuntu.com/dev/lts/serverguide/.htaccess: RewriteRule: cannot compile regular expression '^(*\\.html)\\.*$', referer: http://my-test-website/~doug/linux/ubuntu-docs/help.ubuntu.com/dev/index.html
[Thu Jun 20 14:19:27.360334 2019] [core:alert] [pid 16079] [client 192.168.111.101:63908] /home/doug/public_html/linux/ubuntu-docs/help.ubuntu.com/dev/lts/serverguide/.htaccess: RewriteRule: cannot compile regular expression '^(*\\.html)\\.*$', referer: http://my-test-website/~doug/linux/ubuntu-docs/help.ubuntu.com/dev/index.html

Notice that current attempts are with a wildcard for the language extension. If that is not possible, then the language list is:

ace ar ast be bg bn bs ca cs da de el en en_AU en_CA en_GB eo es et eu fa fi fr gl gu he hr hu id is it ja km ko ku lo lt lv mk ms nb nl oc pl ps pt_BR pt ro ru sk sl sq sr sv th tl tr ug uk ur vi zh_CN zh_TW

Can someone help with this?

The Ubuntu desktop help documents and Ubuntu Serverguide are published on the help.ubuntu.com web site in approximately 60 languages (although many translations are very incomplete). The document compile work flow is somewhat wasteful, because it makes specific language versions of every file, even if they are identical for all languages. For example:

doug@s15:~/docs-trunk/z/html/ubuntu-docs$ ls -l *yelp-note-warning*
-rw-r--r-- 1 doug doug 1088 Mar 13 23:55 yelp-note-warning.png.am
-rw-r--r-- 1 doug doug 1088 Mar 13 23:55 yelp-note-warning.png.ar
-rw-r--r-- 1 doug doug 1088 Mar 13 23:55 yelp-note-warning.png.ast
-rw-r--r-- 1 doug doug 1088 Mar 13 23:55 yelp-note-warning.png.az
...
-rw-r--r-- 1 doug doug 1088 Mar 13 23:55 yelp-note-warning.png.ur
-rw-r--r-- 1 doug doug 1088 Mar 13 23:55 yelp-note-warning.png.uz
-rw-r--r-- 1 doug doug 1088 Mar 13 23:55 yelp-note-warning.png.zh-CN
-rw-r--r-- 1 doug doug 1088 Mar 13 23:55 yelp-note-warning.png.zh-HK
-rw-r--r-- 1 doug doug 1088 Mar 13 23:55 yelp-note-warning.png.zh-TW

For such a file it would be adequate to just have one file called yelp-note-warning.png

For completeness, an example where we would not be able to make just one file:

doug@s15:~/docs-trunk/z/html/ubuntu-docs$ ls -l figures/unity-workspace-intro*
-rw-r--r-- 1 doug doug 48077 Mar 13 23:55 figures/unity-workspace-intro.png.am
-rw-r--r-- 1 doug doug 48077 Mar 13 23:55 figures/unity-workspace-intro.png.ar
-rw-r--r-- 1 doug doug 48077 Mar 13 23:55 figures/unity-workspace-intro.png.ast
...
-rw-r--r-- 1 doug doug 48077 Mar 13 23:55 figures/unity-workspace-intro.png.da
-rw-r--r-- 1 doug doug 64335 Mar 13 23:55 figures/unity-workspace-intro.png.de
-rw-r--r-- 1 doug doug 48077 Mar 13 23:55 figures/unity-workspace-intro.png.el
...
-rw-r--r-- 1 doug doug 48077 Mar 13 23:55 figures/unity-workspace-intro.png.id
-rw-r--r-- 1 doug doug 47152 Mar 13 23:55 figures/unity-workspace-intro.png.it
-rw-r--r-- 1 doug doug 48077 Mar 13 23:55 figures/unity-workspace-intro.png.ja
...
-rw-r--r-- 1 doug doug 48077 Mar 13 23:55 figures/unity-workspace-intro.png.zh-HK
-rw-r--r-- 1 doug doug 48077 Mar 13 23:55 figures/unity-workspace-intro.png.zh-TW

My question is: How can we identify that all files are identical and if so replace them with one file without a language specific suffix?

Pseudo code:

For all files, including sub-folders{
   If all language specific versions of the file are identical{
      Replace the language specific versions with one non-language specific version.
   }
}

At this time, we are only concerned with the desktop help docs, although if something is figured out, we would also do it for the Serverguide.

This could be done in the Makefile, if possible (and preferred). Or as a stand alone script (which I suppose could be called from the Makefile). My concern with writing a c program to it, it that not all Ubuntu documentation team members are also c programmers, causing a potential maintenance concern down the road.
It is envisioned, but not a requirement, that the new code, as per the pseudo code above would be added at the very end of the install section of the Makefile:

# Installs all HTML files to a single multilingual directory for subsequent copying to
# the web server document structure (e.g. to run with Apache and MultiViews enabled)
install:
        rm -Rf "$(INSTALLDIR)"/*; \
        mkdir -p "$(INSTALLDIR)"; \
        cp -R "$(HTMLDESTDIR)/"* "$(INSTALLDIR)"; \
        for lc in C $(help_linguas); do \
                lang=`echo $$lc | $(SED) -e 's/[@_]/-/'`; \
                if test "$$lang" = "C"; then lang=en; fi; \
                if test "$$lang" = "gl"; then lang=gl-GL; fi; \
                if test "$$lang" = "ms"; then lang=ms-MS; fi; \
                if test "$$lang" = "pl"; then lang=pl-PL; fi; \
                cp -af "$(INSTALLDIR)/$$lang"/*.css "$(INSTALLDIR)"; \
                rm -Rf "$(INSTALLDIR)/$$lang"/*.css ; \
                find "$(INSTALLDIR)/$$lang" -type f -exec mv {} {}.$$lang \; ; \
                cp -af "$(INSTALLDIR)/$$lang"/* "$(INSTALLDIR)"; \
                rm -Rf "$(INSTALLDIR)/$$lang" ; \
        done
        ... new code, per this question, goes here ...

EDIT: Gunnar's solution is great, but there are still other files that are redundant. All.js files for example, and still some .png files in the main directory.

EDIT: Gunnar's revised solution addresses all the issues, for a net savings of 3326 less files for just the 17.04 desktop docs web pages alone.

References:
The html compile Makefile. (see the install segment at the end.)
The entire project code.
The build procedure.
The web site .htaccess file. Gunnar's answer relies on the language fallback.

For official Ubuntu documentation where the source English files are in docbook xml, there is a requirement of ASCII only characters. We use a "checker" command line (see here).

grep --color='auto' -P -n "[\x80-\xFF]" *.xml

However, the command has a flaw, apparently not on all computers, it misses some lines with non-ASCII characters, potentially resulting in a false O.K. result.

Does anyone have a better suggestion for a ASCII checker command line?

Interested persons might consider to use this file (text file, not a docbook xml file) as a test case. The first three lines with non ASCII characters are lines 9, 14 and 18. Lines 14 and 18 were missed in the check:

$ grep --color='auto' -P -n "[\x80-\xFF]" install.en.txt | head -13
9:Appendix F, GNU General Public License.
330:when things go wrong. The Installation Howto can be found in Appendix A,
337:Chapter 1. Welcome to Ubuntu
359:1.1. What is Ubuntu?
394:1.1.1. Sponsorship by Canonical
402:1.2. What is Debian?
456:1.2.1. Ubuntu and Debian
461:1.2.1.1. Package selection
475:1.2.1.2. Releases
501:1.2.1.3. Development community
520:1.2.1.4. Freedom and Philosophy
534:1.2.1.5. Ubuntu and other Debian derivatives
555:1.3. What is GNU/Linux?

Almost constantly, I use kernels from the Ubuntu kernel PPA, or kernels I have compiled myself, using the Ubuntu kernel configuration.

The problem is when I am using the stock kernel it seems to default to using predictable network interface names (i.e. "p4p1"), but when I use any alternate kernel it seems to default to using the old method (i.e. "eth0").

While not really relevant to this question, I had a hard disk failure on my test computer and am doing a new install of Ubuntu Server 14.04.2.

I have not been able to determine what is the difference between the kernels. I suspect a kernel configuration parameter, but have been unable to identify which one.

My question: How can I make predictable network interface names work consistently across all kernels?

Additional notes: Somewhere I saw to disable predictable network interface names to add this to grub:

GRUB_CMDLINE_LINUX_DEFAULT="net.ifnames=0 biosdevname=0"

so, I thought the opposite might help:

GRUB_CMDLINE_LINUX_DEFAULT="net.ifnames=1 biosdevname=1"

but it made no difference. Actually, I can never get the non-stock kernel to use biosdevname, no matter the setting. And I can never get the stock kernel to generate a /etc/udev/rules.d/70-persistent-net.rules file (even though, that is not what I want), no matter the setting of net.ifnames. If I have net.ifnames=1, then at least the non-stock kernel doesn't generate an incorrect /etc/udev/rules.d/70-persistent-net.rules file.

Excerpts from /var/log/udev:
stock kernel:

KERNEL[9.216730] add      /devices/pci0000:00/0000:00:1c.4/0000:03:00.0/net/p4p1 (net)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:1c.4/0000:03:00.0/net/p4p1
IFINDEX=2
INTERFACE=p4p1
SEQNUM=1945
SUBSYSTEM=net
...
UDEV  [9.241073] add      /devices/pci0000:00/0000:00:1c.4/0000:03:00.0/net/p4p1 (net)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:1c.4/0000:03:00.0/net/p4p1
ID_BUS=pci
ID_MODEL_FROM_DATABASE=P8P67 and other motherboards
ID_MODEL_ID=0x8168
ID_NET_NAME_MAC=enxf46d04652d8e
ID_NET_NAME_PATH=enp3s0
ID_OUI_FROM_DATABASE=ASUSTek COMPUTER INC.
ID_PCI_CLASS_FROM_DATABASE=Network controller
ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
ID_VENDOR_FROM_DATABASE=Realtek Semiconductor Co., Ltd.
ID_VENDOR_ID=0x10ec
IFINDEX=2
INTERFACE=p4p1

From Ubuntu PPA kernel 4.1RC5:

KERNEL[10.772566] add      /devices/pci0000:00/0000:00:1c.4/0000:03:00.0/net/eth0 (net)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:1c.4/0000:03:00.0/net/eth0
IFINDEX=2
INTERFACE=eth0
SEQNUM=1962
SUBSYSTEM=net
...
UDEV  [11.185866] add      /devices/pci0000:00/0000:00:1c.4/0000:03:00.0/net/eth0 (net)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:1c.4/0000:03:00.0/net/eth0
ID_BUS=pci
ID_MODEL_FROM_DATABASE=P8P67 and other motherboards
ID_MODEL_ID=0x8168
ID_NET_NAME_MAC=enxf46d04652d8e
ID_NET_NAME_PATH=enp3s0
ID_OUI_FROM_DATABASE=ASUSTek COMPUTER INC.
ID_PCI_CLASS_FROM_DATABASE=Network controller
ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
ID_VENDOR_FROM_DATABASE=Realtek Semiconductor Co., Ltd.
ID_VENDOR_ID=0x10ec
IFINDEX=2
INTERFACE=eth0
SEQNUM=1962
SUBSYSTEM=net
USEC_INITIALIZED=3227

Note: I have temporarily moved past this issue by allowing a /etc/udev/rules.d/70-persistent-net.rules file to be created, and then editing it for proper device names. Ultimately, I would like to get rid of the file.