Home / server / Questions / 1033625
In Process
Alex
Asked: 2020-09-13 01:58:05 +0800 CST

Adding a VLAN (over bond/lacp) interface to bridge makes physical interfaces to go down

  • 772

I have a working configuration of a team interface bonding two physical interfaces, with several VLANs.

ens1f1 -----               |--- team0.1826 (172.18.x.y/28)
           |--- team0 -----|--- team0.1651
ens2f0 -----               |--- team9.1655

I need to start a kvm virtual machine on this host, on the vlan 1826 network. So I am trying to add a bridge including the team0.1826 interface in the following manner:

  • remove the previous vlan interface
nmcli con del team0.1826
  • Add a bridge interface
nmcli connection add type bridge con-name br0 ifname br0 \
        ipv4.method manual ipv4.addresses '172.18.x.y/28' ipv4.gateway '172.18.x.z'
  • Recreate the vlan interface, as a slave to the bridge
nmcli connection add type vlan con-name team01826 ifname team0.1826 \
        dev team0 id 1826 master br0 slave-type bridge

It happens that as soon as I run this last command, all the interfaces are going down, including the physical ones. I am guessing that doing this somehow breaks the LACP communication to the switch, making the interfaces to be brought down. However I am not understanding what I am doing wrong and how to make things work. As soon as I remove the new bridge-slave team0.1826 interface, interfaces are going UP again.

Some additional information:

  • team0 configuration
$ cat ifcfg-team0
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=team0
UUID=24c8380b-43ca-4977-9dc0-27e27e3ec387
DEVICE=team0
ONBOOT=yes
DEVICETYPE=Team
TEAM_CONFIG="{\"device\": \"team0\", \"runner\": {\"name\": \"lacp\", \"fast_rate\": true, \"tx_hash\": [\"eth\", \"ipv4\", \"ipv6\"]}, \"link_watch\": {\"name\": \"ethtool\"}, \"ports\": {\"ens1f1\": {}, \"ens2f0\": {}}}"

  • I have tried with and without STP on the br0 interface, in case of conflict but no change.

  • With "debug_level: 2" in team configuration, and with network manager in debug level, I have those logs, but it seems there is nothing relevant in them:

Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5499] ifcfg-rh: add connection /etc/sysconfig/network-scripts/ifcfg-team0.1826 (c5726afe-2458-492c-9089-f6828559ab9b,"team0.1826")
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5504] manager: (team0.1826): new VLAN device (/org/freedesktop/NetworkManager/Devices/33)
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5520] audit: op="connection-add" uuid="c5726afe-2458-492c-9089-f6828559ab9b" name="team0.1826" pid=1374856 uid=0 result="success"
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5585] device (team0.1826): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5728] device (team0.1826): carrier: link connected
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5744] device (team0.1826): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5799] policy: auto-activating connection 'team0.1826' (c5726afe-2458-492c-9089-f6828559ab9b)
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5805] device (team0.1826): Activation: starting connection 'team0.1826' (c5726afe-2458-492c-9089-f6828559ab9b)
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5806] device (team0.1826): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5811] device (team0.1826): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5836] device (team0.1826): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5923] device (br0): attached bridge port team0.1826
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5923] device (team0.1826): Activation: connection 'team0.1826' enslaved, continuing activation
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5925] device (br0): IPv4 config waiting until carrier is on
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5925] device (br0): IPv6 config waiting until carrier is on
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5926] device (team0.1826): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5933] device (team0.1826): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5935] device (team0.1826): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Sep 11 16:26:39 hostname NetworkManager[1059]: <info>  [1599834399.5947] device (team0.1826): Activation: successful, device activated.
Sep 11 16:26:41 hostname NetworkManager[1059]: ens1f1: Changed port state: "current" -> "disabled"
Sep 11 16:26:41 hostname NetworkManager[1059]: ens1f1: ethtool-link went down.
Sep 11 16:26:41 hostname NetworkManager[1059]: ens2f0: Changed port state: "current" -> "disabled"
Sep 11 16:26:41 hostname NetworkManager[1059]: carrier changed to DOWN
Sep 11 16:26:41 hostname NetworkManager[1059]: ens2f0: ethtool-link went down.
Sep 11 16:26:47 hostname NetworkManager[1059]: <info>  [1599834407.5362] device (team0.1655): state change: activated -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')
Sep 11 16:26:47 hostname NetworkManager[1059]: <info>  [1599834407.5503] device (team0.1659): state change: activated -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')
Sep 11 16:26:47 hostname NetworkManager[1059]: <info>  [1599834407.5655] manager: NetworkManager state is now CONNECTING
Sep 11 16:26:47 hostname NetworkManager[1059]: <info>  [1599834407.5668] manager: NetworkManager state is now CONNECTED_GLOBAL
Sep 11 16:26:47 hostname NetworkManager[1059]: <info>  [1599834407.5669] policy: set 'team0.1661' (team0.1661) as default for IPv4 routing and DNS
Sep 11 16:26:47 hostname NetworkManager[1059]: <info>  [1599834407.5676] device (team0.1661): state change: activated -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')
Sep 11 16:26:47 hostname NetworkManager[1059]: <info>  [1599834407.5825] manager: NetworkManager state is now CONNECTING
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.7929] agent-manager: req[0x55ac4d810020, :1.6180426/nmcli-connect/0]: agent registered
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.7941] device (br0): state change: ip-config -> deactivating (reason 'new-activation', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.7944] manager: NetworkManager state is now CONNECTED_LOCAL
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.7947] device (br0): disconnecting for new activation request.
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.7947] audit: op="connection-activate" uuid="42451cd5-641a-45cf-8c4f-be4dc4227136" name="br0" pid=1377109 uid=0 result="success"
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.7947] device (br0): state change: deactivating -> disconnected (reason 'new-activation', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8127] device (br0): detached bridge port team0.1826
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8143] device (br0): Activation: starting connection 'br0' (42451cd5-641a-45cf-8c4f-be4dc4227136)
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8149] device (team0.1826): state change: activated -> disconnected (reason 'new-activation', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8202] device (br0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8205] manager: NetworkManager state is now CONNECTING
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8373] device (team0.1826): state change: disconnected -> unmanaged (reason 'user-requested', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8377] policy: auto-activating connection 'team0.1826' (c5726afe-2458-492c-9089-f6828559ab9b)
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8379] device (br0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8401] device (team0.1826): state change: unmanaged -> unavailable (reason 'user-requested', sys-iface-state: 'external')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8408] device (team0.1826): state change: unavailable -> disconnected (reason 'user-requested', sys-iface-state: 'external')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8414] device (team0.1826): Activation: starting connection 'team0.1826' (c5726afe-2458-492c-9089-f6828559ab9b)
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8414] device (team0.1826): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8419] device (br0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8421] device (br0): IPv4 config waiting until carrier is on
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8421] device (br0): IPv6 config waiting until carrier is on
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8423] device (team0.1826): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8568] device (team0.1826): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8793] device (br0): attached bridge port team0.1826
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8793] device (team0.1826): Activation: connection 'team0.1826' enslaved, continuing activation
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8794] device (br0): IPv4 config waiting until carrier is on
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8795] device (br0): IPv6 config waiting until carrier is on
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8796] device (team0.1826): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8802] device (team0.1826): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8803] device (team0.1826): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Sep 11 16:28:26 hostname NetworkManager[1059]: <info>  [1599834506.8814] device (team0.1826): Activation: successful, device activated.
Sep 11 16:29:21 hostname NetworkManager[1059]: <info>  [1599834561.1835] device (team0.1826): state change: activated -> deactivating (reason 'connection-removed', sys-iface-state: 'managed')
Sep 11 16:29:21 hostname NetworkManager[1059]: <info>  [1599834561.1987] device (br0): detached bridge port team0.1826
Sep 11 16:29:21 hostname NetworkManager[1059]: <info>  [1599834561.1987] device (team0.1826): released from master device br0
Sep 11 16:29:21 hostname NetworkManager[1059]: <info>  [1599834561.1999] audit: op="connection-delete" uuid="c5726afe-2458-492c-9089-f6828559ab9b" name="team0.1826" pid=1378747 uid=0 result="success"
Sep 11 16:29:21 hostname NetworkManager[1059]: <info>  [1599834561.2000] device (team0.1826): state change: deactivating -> disconnected (reason 'connection-removed', sys-iface-state: 'managed')
Sep 11 16:29:21 hostname NetworkManager[1059]: <info>  [1599834561.2243] device (team0.1826): state change: disconnected -> unmanaged (reason 'user-requested', sys-iface-state: 'managed')
Sep 11 16:31:41 hostname NetworkManager[1059]: ens1f1: Changed port state: "disabled" -> "expired"
Sep 11 16:31:41 hostname NetworkManager[1059]: ens1f1: ethtool-link went up.
Sep 11 16:31:41 hostname NetworkManager[1059]: <info>  [1599834701.6413] device (ens1f1): carrier: link connected
Sep 11 16:31:41 hostname NetworkManager[1059]: ens2f0: Changed port state: "disabled" -> "expired"
Sep 11 16:31:41 hostname NetworkManager[1059]: ens2f0: ethtool-link went up.
Sep 11 16:31:41 hostname NetworkManager[1059]: <info>  [1599834701.9899] device (ens2f0): carrier: link connected
Sep 11 16:31:42 hostname NetworkManager[1059]: ens1f1: Changed port state: "expired" -> "current"
Sep 11 16:31:42 hostname NetworkManager[1059]: carrier changed to UP
Sep 11 16:31:42 hostname NetworkManager[1059]: <info>  [1599834702.6431] device (team0): carrier: link connected
Sep 11 16:31:42 hostname NetworkManager[1059]: <info>  [1599834702.6434] device (team0.1655): carrier: link connected
Sep 11 16:31:42 hostname NetworkManager[1059]: <info>  [1599834702.6436] device (team0.1659): carrier: link connected
Sep 11 16:31:42 hostname NetworkManager[1059]: <info>  [1599834702.6438] device (team0.1661): carrier: link connected
Sep 11 16:31:42 hostname NetworkManager[1059]: <info>  [1599834702.6439] device (team0.1655): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
Sep 11 16:31:42 hostname NetworkManager[1059]: <info>  [1599834702.6502] device (team0.1659): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
Sep 11 16:31:42 hostname NetworkManager[1059]: <info>  [1599834702.6564] device (team0.1661): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')

Is anyone having an idea on how this happens, and how to make it work?

centos vlan bonding bridge linux-networking

1 Answers

  1. It happens that I managed to mount the bridge, by following exactly those steps:

    • Marking every connection as down
    nmcli con down team0.1826
nmcli con down team0.1651
nmcli con down team0.1655
nmcli con down team0
nmcli con down ens1f1
nmcli con down ens2f0

    Point was: no interface up before starting.

    • recreating the vlan connection without the ips
    nmcli con del team0.1826
nmcli con add type vlan con-name team0.1826 ifname team0.1826 dev team0 id 1826
    • creating the bridge. I disabled STP but I don't think it has any incidence, just seen this a lot on KVM tutorials.
    nmcli connection add type bridge con-name br0 ifname br0 bridge.stp no
    • tying the bridge and the vlan together
    nmcli con mod team0.1826 connection.master br0 connection.slave-type bridge
    • configuring the bridge
    nmcli con mod br0 ipv4.method manual ipv4.addresses '172.18.x.y/28' ipv4.gateway '172.18.x.z'
    • starting everything
    nmcli con up br0
nmcli con up team0.1826
nmcli con up team0
nmcli con up ens1f1
nmcli con up ens2f0
nmcli con up team0.1651
nmcli con up team0.1655

    And voilà! the bridge did go up.

    I think that NetworkManager may be a little picky. I strongly suspect that splitting team0.1826 interface into creation and tying to bridge steps did the trick.

    • 0