John Karasev's questions -server

John Karasev

Asked: 2021-09-28 20:58:17 +0800 CST

default route does not work

0

root@kolla02:~/dev-env# ip r s
default via 192.168.1.101 dev eno1 
10.0.0.0/16 dev eno2 proto kernel scope link src 10.0.0.1 
10.1.0.0/16 dev virbr2 proto kernel scope link src 10.1.0.1 
10.230.0.0/16 dev virbr3 proto kernel scope link src 10.230.0.1 
192.168.0.0/16 dev eno1 proto kernel scope link src 192.168.20.11 
192.168.1.101 dev eno1 proto dhcp scope link src 192.168.20.11 metric 100 
192.168.121.0/24 dev virbr1 proto kernel scope link src 192.168.121.1 
root@kolla02:~/dev-env# curl google.com
curl: (7) Failed to connect to <proxy> port <port>: No route to host
root@kolla02:~/dev-env# curl google.com --interface eno1
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

I need the curl to work without specifying the interface every time. The default route is set so not sure why it does not work without the interface. How is this fixed?

OS --> Ubuntu 20.04

If all the extra routes get removed to leave this:

default via 192.168.1.101 dev eno1 proto dhcp src 192.168.20.11 metric 100 
192.168.0.0/16 dev eno1 proto kernel scope link src 192.168.20.11 
192.168.1.101 dev eno1 proto dhcp scope link src 192.168.20.11 metric 100

The default route starts to work again:

root@kolla02:~/dev-env# curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML

But those routes are needed for vagrant and is not a solution in this case.

UPDATE:
seems like the conflicting entry is:

10.1.0.0/16 dev virbr2 proto kernel scope link src 10.1.0.1

as soon as this is removed, everything works fine. But this route is needed for vagrant to work.

root@kolla02:~/dev-env# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a4:bf:01:55:f6:28 brd ff:ff:ff:ff:ff:ff
    inet 192.168.20.11/16 brd 192.168.255.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::a6bf:1ff:fe55:f628/64 scope link 
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a4:bf:01:55:f6:29 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a6bf:1ff:fe55:f629/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:0b:2d:4f:ee brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:07:f9:55 brd ff:ff:ff:ff:ff:ff
    inet 192.168.121.1/24 brd 192.168.121.255 scope global virbr1
       valid_lft forever preferred_lft forever
6: virbr1-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr1 state DOWN group default qlen 1000
    link/ether 52:54:00:07:f9:55 brd ff:ff:ff:ff:ff:ff
7: virbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:9c:15:16 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/16 brd 10.1.255.255 scope global virbr2
       valid_lft forever preferred_lft forever
8: virbr2-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr2 state DOWN group default qlen 1000
    link/ether 52:54:00:9c:15:16 brd ff:ff:ff:ff:ff:ff
9: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:99:a4:2f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe99:a42f/64 scope link 
       valid_lft forever preferred_lft forever
10: macvtap0@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:2e:82:8b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe2e:828b/64 scope link 
       valid_lft forever preferred_lft forever
11: macvtap1@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:ff:40:a0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:feff:40a0/64 scope link 
       valid_lft forever preferred_lft forever
12: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr2 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:44:07:25 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe44:725/64 scope link 
       valid_lft forever preferred_lft forever
13: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:55:fc:18 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe55:fc18/64 scope link 
       valid_lft forever preferred_lft forever
14: macvtap2@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:ee:cf:c6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:feee:cfc6/64 scope link 
       valid_lft forever preferred_lft forever
15: macvtap3@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:b9:1a:e0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:feb9:1ae0/64 scope link 
       valid_lft forever preferred_lft forever
16: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr2 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:65:ab:6f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe65:ab6f/64 scope link 
       valid_lft forever preferred_lft forever
17: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:0f:8e:63 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe0f:8e63/64 scope link 
       valid_lft forever preferred_lft forever
18: macvtap4@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:2c:39:9f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe2c:399f/64 scope link 
       valid_lft forever preferred_lft forever
19: macvtap5@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:78:8b:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe78:8be2/64 scope link 
       valid_lft forever preferred_lft forever
20: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr2 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:3d:c3:28 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe3d:c328/64 scope link 
       valid_lft forever preferred_lft forever
21: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:bd:b4:dd brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:febd:b4dd/64 scope link 
       valid_lft forever preferred_lft forever
22: macvtap6@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:a3:7a:db brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fea3:7adb/64 scope link 
       valid_lft forever preferred_lft forever
23: macvtap7@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:9a:31:0f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe9a:310f/64 scope link 
       valid_lft forever preferred_lft forever
24: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr2 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:78:db:b6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe78:dbb6/64 scope link 
       valid_lft forever preferred_lft forever
25: vnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:93:71:45 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe93:7145/64 scope link 
       valid_lft forever preferred_lft forever
26: macvtap8@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:99:c1:9c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe99:c19c/64 scope link 
       valid_lft forever preferred_lft forever
27: macvtap9@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:11:a9:ab brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe11:a9ab/64 scope link 
       valid_lft forever preferred_lft forever
28: vnet9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr2 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:37:be:e0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe37:bee0/64 scope link 
       valid_lft forever preferred_lft forever
29: vnet10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:78:6f:36 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe78:6f36/64 scope link 
       valid_lft forever preferred_lft forever
30: macvtap10@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:51:1e:a4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe51:1ea4/64 scope link 
       valid_lft forever preferred_lft forever
31: macvtap11@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:7f:f8:6e brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe7f:f86e/64 scope link 
       valid_lft forever preferred_lft forever
32: vnet11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr2 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:98:1d:35 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe98:1d35/64 scope link 
       valid_lft forever preferred_lft forever

UPDATE2: I changed the vagrant private network from 10.1.0.0/16 to a private range 172.16.0.0/16 and it started to work. @Michael Hampton gave this hint of conflicting routes. The http proxy was rerouted to the internal vagrant private network instead of going out.

Vagrant.configure('2') do |config|
  (1..6).each do |i|
    config.vm.define "vm-#{i}" do |node|
      config.vm.box = 'generic/ubuntu2004'
      node.vm.network 'public_network',
                      :dev => 'eno1',
                      :mode => 'bridge',
                      :netmask => '255.255.0.0',
                      :ip => "192.168.20.#{50 + i}"
      node.vm.network 'public_network',
                      :dev => 'eno2',
                      :bridge => 'eno2',
                      :netmask => '255.255.0.0',
                      :ip => "10.0.87.#{i}"
      node.vm.network 'private_network', 
                      :forward_mode => 'nat', 
                      :forward_device => 'eno1',
                      :ip => "172.16.0.#{10+i}",
                      :netmask => '255.255.0.0'
      node.vm.provider 'libvirt' do |v|  # docs -> https://github.com/vagrant-libvirt/vagrant-libvirt
        # memory in MBytes
        v.memory = $mem
        # number of virtual cpus.
        v.cpus = $cpus
        # nested virtualization
        v.nested = true
        # boot image size in GB
        v.machine_virtual_size = 150
        v.storage :file,
                  :size => 100,
                  :allow_existing => true,
                  :path => "vm-#{i}-data.img"
      node.vm.provision "shell", inline: $script
      end
    end
  end
end

John Karasev

Asked: 2021-04-29 22:32:10 +0800 CST

veth does not respond to other veth ping requests on same bridge

0

I created two veth pairs and connected them to the same bridge. However, they do not respond to ping requests. This is ubuntu 20.04.

# create the interfaces and bridge  
sudo ip link add br0 type bridge
sudo ip link add i0 type veth peer name i0-p
sudo ip link add i1 type veth peer name i1-p

# connect veth peers into bridge as well as physical interface 
sudo ip link set eno2 master br0
sudo ip link set i0-p master br0
sudo ip link set i1-p master br0

# bring up all the interfaces/bridges up. 
sudo ip link set br0 up
sudo ip link set eno2 up
sudo ip link set i0-p up 
sudo ip link set i1-p up 
sudo ip link set i0 up 
sudo ip link set i1 up 

# set the addresses 
sudo ip a add 10.0.10.4/16 dev br0
sudo ip a add 10.0.10.5/16 dev i0
sudo ip a add 10.0.10.6/16 dev i1

                                                                                   
                                                                                   
                                             +-------------+                       
                                             |             |                       
         +---------+               +---------|             |                       
         |         |               |         |             |                       
         |  i0     |---------------|  i0-p   |             |                       
         |         |               |         |             |                       
         |10.0.10.5/16             |         |             |            +---------+
         |         |               |         |             |            |         |
         +---------+               +---------|             |            |  en02   |
                                             |     br0     |------------|         |
                                             |             |            |         |
                                             |10.0.10.4/16 |            |         |
         +---------+               +---------+             |            +---------+
         |         |               |         |             |                       
         |  i1     |---------------|  i1-p   |             |                       
         |         |               |         |             |                       
         |10.0.10.6/16             |         |             |                       
         |         |               |         |             |                       
         +---------+               +---------+             |                       
                                             |             |                       
                                             +-------------+

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a4:bf:01:55:f3:a4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.20.3/16 brd 192.168.255.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::a6bf:1ff:fe55:f3a4/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether a4:bf:01:55:f3:a5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a6bf:1ff:fe55:f3a5/64 scope link
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:09:12:3c:7c brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
10: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 32:b1:3c:cd:ab:b0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.10.4/16 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::30b1:3cff:fecd:abb0/64 scope link
       valid_lft forever preferred_lft forever
11: i0-p@i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 32:b1:3c:cd:ab:b0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30b1:3cff:fecd:abb0/64 scope link
       valid_lft forever preferred_lft forever
12: i0@i0-p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ba:04:42:b1:62:f8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.10.5/16 scope global i0
       valid_lft forever preferred_lft forever
    inet6 fe80::b804:42ff:feb1:62f8/64 scope link
       valid_lft forever preferred_lft forever
13: i1-p@i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 72:b0:ab:ca:c4:55 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::70b0:abff:feca:c455/64 scope link
       valid_lft forever preferred_lft forever
14: i1@i1-p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:47:29:14:84:bb brd ff:ff:ff:ff:ff:ff
    inet 10.0.10.6/16 scope global i1
       valid_lft forever preferred_lft forever
    inet6 fe80::f447:29ff:fe14:84bb/64 scope link
       valid_lft forever preferred_lft forever

# sysctl -p
net.ipv4.ip_forward = 1

# ping 10.0.10.6 -I i0
PING 10.0.10.6 (10.0.10.6) from 10.0.10.5 i0: 56(84) bytes of data.
^C
--- 10.0.10.6 ping statistics ---
14 packets transmitted, 0 received, 100% packet loss, time 13296ms

# tcpdump -i i0 not stp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on i0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:25:12.616946 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:13.625187 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:14.649177 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:15.673372 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:16.697226 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:17.721230 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:18.745363 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:19.769271 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:20.793229 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:21.817332 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:22.841256 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:23.865261 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:24.889354 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:25.913245 ARP, Request who-has kolla01 tell kolla01, length 28
06:25:26.937169 ARP, Request who-has kolla01 tell kolla01, length 28

Anyone have any suggestions?

UPDATE:
applying suggestions made by kab00m:

# tcpdump -ni lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
15:59:00.976656 IP 10.0.10.2 > 10.0.10.2: ICMP host 10.0.10.3 unreachable, length 92
15:59:00.976667 IP 10.0.10.2 > 10.0.10.2: ICMP host 10.0.10.3 unreachable, length 92
15:59:00.976674 IP 10.0.10.2 > 10.0.10.2: ICMP host 10.0.10.3 unreachable, length 92
15:59:04.080688 IP 10.0.10.2 > 10.0.10.2: ICMP host 10.0.10.3 unreachable, length 92
15:59:04.080698 IP 10.0.10.2 > 10.0.10.2: ICMP host 10.0.10.3 unreachable, length 92
15:59:04.080705 IP 10.0.10.2 > 10.0.10.2: ICMP host 10.0.10.3 unreachable, length 92
15:59:07.152677 IP 10.0.10.2 > 10.0.10.2: ICMP host 10.0.10.3 unreachable, length 92
^C
7 packets captured
14 packets received by filter
0 packets dropped by kernel

# destroy everything 
sudo ip link delete i0
sudo ip link delete i1
sudo ip link delete br0 

# create namespaces 
ip netns add net0 
ip netns add net1 

# create veth pairs and bridge 
ip link add i0 type veth peer name i0-p
ip link add i1 type veth peer name i1-p
ip link add br0 type bridge

# add devices to network namespace 
ip link set i0 netns net0
ip link set i1 netns net1

# add second peers into bridge 
ip link set eno2 master br0
ip link set i0-p master br0
ip link set i1-p master br0

# add addresses 
ip a add 10.0.10.4/16 dev br0
ip netns exec net0 ip a add 10.0.10.5/16 dev i0 
ip netns exec net1 ip a add 10.0.10.6/16 dev i1

# bring up all devices 
ip netns exec net0 ip link set i0 up
ip netns exec net1 ip link set i1 up
ip link set i0-p up
ip link set i1-p up
ip link set br0 up
ip link set eno2 up

# ip netns exec net0 ping 10.0.10.6 -I i0
PING 10.0.10.6 (10.0.10.6) from 10.0.10.5 i0: 56(84) bytes of data.
64 bytes from 10.0.10.6: icmp_seq=1 ttl=64 time=0.058 ms
64 bytes from 10.0.10.6: icmp_seq=2 ttl=64 time=0.051 ms
64 bytes from 10.0.10.6: icmp_seq=3 ttl=64 time=0.051 ms
^C
--- 10.0.10.6 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2046ms
rtt min/avg/max/mdev = 0.051/0.053/0.058/0.003 ms

John Karasev

Asked: 2020-12-18 16:18:37 +0800 CST

How to provision baremetal kuberenetes cluster using magnum with only one vlan?

3

There does not seem to be a guide that follows through on how to deploy a baremetal Kubernetes cluster using magnum. I got to the point of having the baremetal servers turn on and initiate the pxe request, however, the dnsmasq server does not respond for BOOTP requests. What are the required steps for this to work?

Update:
Not sure who closed this question. If there is something off topic please let me know.

Magnum was able to initiate the pxe requests after setting the fixed-network to the baremetal provisioning network. However, the external-network flag is also required and there is only one vlan in this setup (which has been setup as the bm-provision network). I attempted to create a public network like this (without any underlying physical network devices):

openstack network create public --provider-network-type vxlan \
                                  --external \
                                  --project service

openstack subnet create public-subnet --network public \
                                  --subnet-range 172.16.10.0/24 \
                                  --gateway 172.16.10.1 \
                                  --ip-version 4 

openstack coe cluster template create bmt \
--image fa27 \
--keypair mykey \
--external-network public \
--fixed-network bm-provision \
--fixed-subnet bm-provision-subnet \
--master-flavor bm.dev \
--flavor bm.dev \
--network-driver calico \
--coe kubernetes

openstack coe cluster create bm \
                        --cluster-template bmt \
                        --master-count 1 \
                        --node-count 1 \
                        --keypair mykey

And this provisions the baremetal with the os over PXE via ironic, however, run into this:

{
  "default-master": "Resource CREATE failed: NotFound: resources.kube_masters.resources[0].resources.kube_master_floating: External network fa5174bb-d01d-48ca-a564-bbff283b1141 is not reachable from subnet 3e68266a-e28b-4d2c-8cd6-4042ac5a38ac.  Therefore, cannot associate Port 4950a517-31c6-4ed2-b7f8-03c3286063b3 with a Floating IP.\nNeutron server returns request_ids: ['req-bc587047-bf37-49ba-a46c-4d299f85812b']",
  "default-worker": "Resource CREATE failed: NotFound: resources.kube_masters.resources[0].resources.kube_master_floating: External network fa5174bb-d01d-48ca-a564-bbff283b1141 is not reachable from subnet 3e68266a-e28b-4d2c-8cd6-4042ac5a38ac.  Therefore, cannot associate Port 4950a517-31c6-4ed2-b7f8-03c3286063b3 with a Floating IP.\nNeutron server returns request_ids: ['req-bc587047-bf37-49ba-a46c-4d299f85812b']"
}

For this experiment, I don't need the external network, the bm-provision has a gateway pointing to nat so there is already access to the internet. Trying to achieve this with one VLAN. Will this be possible?

If you need any additional information, please let me know.

Answer
Cannot make a separate answer since this question is closed but here is what was essentially executed:

# add some more interfaces facing into the same vlan
# note macvlan was attempted also but BOOTP requests did no go through for some reason
ip link add kolla_i type veth peer name kolla_b
for i in `seq 1 10`; do ip link add p${i}_i type veth peer name p${i}_b; done
ip link add eno2_br type bridge
ip link set eno2_br up
ip link set eno2 master eno2_br
ip link set kolla_b master eno2_br
ip link set kolla_b up
ip link set kolla_i up 
ip a add 10.0.0.4/16 dev kolla_i
for i in `seq 1 10`; do ip link set p${i}_b master eno2_br; done
for i in `seq 1 10`; do ip link set p${i}_b up; done
for i in `seq 1 10`; do ip link set p${i}_i up; done

in globals.yml file (for kolla-ansible provisioning):

kolla_internal_vip_address: "10.0.0.4"
network_interface: "kolla_i"
neutron_external_interface: "p1_i,p2_i,p3_i,p4_i,p5_i,p6_i,p7_i,p8_i,p9_i,p10_i"
# this option does not exist so just add it into globals.yml
neutron_bridge_name: "br-ex1,br-ex2,br-ex3,br-ex4,br-ex5,br-ex6,br-ex7,br-ex8,br-ex9,br-ex10"
ironic_dnsmasq_interface: "p1_i"
ironic_dnsmasq_dhcp_range: "10.0.2.1,10.0.2.5"

This basically gives you 10 physical network to use within the same vlan. Here is how the ovs conf file looked after installation:

docker exec -it --user root neutron_openvswitch_agent bash -c "cat /etc/neutron/plugins/ml2/openvswitch_agent.ini"

[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

[ovs]
bridge_mappings = physnet1:br-ex1,physnet2:br-ex2,physnet3:br-ex3,physnet4:br-ex4,physnet5:br-ex5,physnet6:br-ex6,physnet7:br-ex7,physnet8:br-ex8,physnet9:br-ex9,physnet10:br-ex10
datapath_type = system
ovsdb_connection = tcp:127.0.0.1:6640
local_ip = 10.0.0.4

The next step was to create the networks:

openstack network create \
--share \
--provider-network-type flat \
--provider-physical-network physnet1 \
--external \
provision

openstack subnet create \
--network provision \
--allocation-pool start=10.0.2.6,end=10.0.2.230 \
--gateway 10.0.0.10 \
--subnet-range 10.0.0.0/16 \
provision-subnet 


openstack network create \
--share \
--provider-network-type flat \
--provider-physical-network physnet2 \
--external \
public

openstack subnet create \
--network public \
--allocation-pool start=10.1.0.1,end=10.1.0.10 \
--allocation-pool start=10.1.0.12,end=10.1.0.250 \
--gateway 10.1.0.11 \
--subnet-range 10.1.0.0/16 \
public-subnet

create the router

openstack router create provision-public
openstack router set provision-public --external-gateway public
openstack router add subnet provision-public provision-subnet

register the bare-metal nodes:

openstack flavor create --ram 1048576 --disk 100 --vcpus 64 bm.dev
openstack flavor set --property baremetal=true bm.dev
openstack flavor set --property resources:CUSTOM_BAREMETAL_DEV=1 bm.dev
openstack flavor set --property resources:VCPU=0 bm.dev
openstack flavor set --property resources:MEMORY_MB=0 bm.dev
openstack flavor set --property resources:DISK_GB=0 bm.dev


openstack baremetal node create --name dev02 \
--driver ipmi \
--driver-info ipmi_username=<user> \
--driver-info ipmi_password=<pass> \
--driver-info ipmi_address=<ipmi_addr> \
--driver-info deploy_kernel=http://10.0.0.4:8089/ironic-agent.kernel \
--driver-info deploy_ramdisk=http://10.0.0.4:8089/ironic-agent.initramfs \
--driver-info cleaning_network=provision \
--driver-info provisioning_network=provision \
--deploy-interface=direct \
--network-interface=flat \
--driver-info force_persistent_boot_device=True \
--property capabilities=boot_mode:uefi \
--property cpu_arch=x86_64 \
--property local_gb=1000 \
--resource-class baremetal.dev

openstack baremetal port create <mac_addr> --node <id>
openstack baremetal node manage dev02
openstack baremetal node provide dev02

openstack baremetal node create --name dev03 \
--driver ipmi \
--driver-info ipmi_username=<user> \
--driver-info ipmi_password=<pass> \
--driver-info ipmi_address=<ipmi_addr> \
--driver-info deploy_kernel=http://10.0.0.4:8089/ironic-agent.kernel \
--driver-info deploy_ramdisk=http://10.0.0.4:8089/ironic-agent.initramfs \
--driver-info cleaning_network=provision \
--driver-info provisioning_network=provision \
--deploy-interface=direct \
--network-interface=flat \
--driver-info force_persistent_boot_device=True \
--property capabilities=boot_mode:uefi \
--property cpu_arch=x86_64 \
--property local_gb=1000 \
--resource-class baremetal.dev

openstack baremetal port create <mac_addr> --node <id>
openstack baremetal node manage dev03 
openstack baremetal node provide dev03

docker exec --user root  nova_conductor bash -c "nova-manage cell_v2 discover_hosts --by-service"

create template and deploy

openstack coe cluster template create bmt \
--image bm \
--keypair mykey \
--external-network public \
--fixed-network provision \
--fixed-subnet provision-subnet \
--master-flavor bm.dev \
--flavor bm.dev \
--network-driver calico \
--coe kubernetes

openstack coe cluster create bm \
                        --cluster-template bmt \
                        --master-count 1 \
                        --node-count 1 \
                        --keypair mykey

Note, this is not for production since all networks will be in the same broadcast domain but suitable for experiments.

default route does not work

veth does not respond to other veth ping requests on same bridge

How to provision baremetal kuberenetes cluster using magnum with only one vlan?

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?