Questions[openstack-nova](server)

I am trying out the volume attachment function at Openstack (version:wallaby) to the server as additional device but failed.

The volume backend is ceph which all of the osds are up and healthy.

ceph-osd/38*                   active    idle   0         172.16.6.64                        Unit is ready (1 OSD)
    ntp/149                      active    idle             172.16.6.64     123/udp            chrony: Ready
ceph-osd/39                    active    idle   1         172.16.6.65                        Unit is ready (1 OSD)
    ntp/147                      active    idle             172.16.6.65     123/udp            chrony: Ready
ceph-osd/40                    active    idle   2         172.16.6.66                        Unit is ready (1 OSD)
    ntp/146*                     active    idle             172.16.6.66     123/udp            chrony: Ready
ceph-osd/41                    active    idle   3         172.16.6.67                        Unit is ready (1 OSD)
    ntp/148                      active    idle             172.16.6.67     123/udp            chrony: Ready

where the servers are served by nova.

The whole volume attachment process is successful on some of the servers. I can attach the volume created to the nova instance as /dev/vdb and /dev/vdc. However, some of the servers cannot. I have checked the /var/log/nova/nova-compute.log and got the below message

 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': error connecting: Invalid argument

Additional info: The volume can be attached when the server is in shutoff state but the server cannot be powered up if the volume is attached. I take a look at the /var/log/nova/nova-compute.log and the ERROR message is as below:

ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: process exited while connecting to monitor: 2021-11-01T16:34:08.889402Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"cinder-ceph","image":"volume-c41ce9db-e375-4b21-920f-e815035b51ed","server":[{"host":"172.16.6.104","port":"6789"},{"host":"172.16.6.106","port":"6789"},{"host":"172.16.6.105","port":"6789"}],"user":"cinder-ceph","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error connecting: Invalid argument

I am pretty sure the volume does not have issue as I can attach the same volume to other nova instance without issue.

I have done the below measure but still no luck:

(1) I recreated the nova instance so with a different instance id but still fail to attach

(2) I have checked the nova and ceph connecting virsh secret and config at /etc/nova/nova.conf and /etc/libvirt/secrets with virsh secret-list. They are the same as other successful volume attachment instance

All of the openstack service is up and running without error. Could anyone give me some clue about the ERROR message?

In Openstack, is it possible to create compute nodes with no hard drives and use PXE in order to boot the host's system and therefore launch instances with no local drive which is needed to boot the VM's image.

If not what's the minimum storage needed to be given to hosts in order to get fully functional system.

In our OpenStack cloud there are a server with server_id and a volume with <volume_id>. We are in a situation where there is a volume that appears to be attached from the perspective of Nova, but not from the perspective of Cinder. There is a record in the nova database for this attachment: MariaDB [(none)]> select * from block_device_mapping where instance_uuid = "<volume_id>", shows the entry.

Nova shows the attachment:

openstack server show <server_id>
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+---------------------------------------------------------
| volumes_attached                    | id='...'                                                 |
|                                     | id='<volume_id'                                          |
+-------------------------------------+----------------------------------------------------------+

Cinder, does not know about this attachment:

openstack volume show <volume_id>
+--------------------------------+--------------------------------------+
| Field                          | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |

From within the VM, the volume seems connected and can be operated on.

I tried setting the volume to an available and detached state, but it is not possible to attach it in any state that I have tried.

It also tried removing the attachment completely. When running openstack server remove volume or nova volume-detach the command finishes without an error and the cinder logging shows: Roll detaching of volume completed successfully. However, the situation does not change.

What would be the best way to recover from this situation? For this specific VM, I think I could remove the database entry from block_device_mapping and rebuild the VM? For the future I would like to know if it is also possible to repair the situation in OpenStack to reflect the situation in the real world. That means, is there any way in which I can add the attachment, so that it is also visible in Cinder?

There does not seem to be a guide that follows through on how to deploy a baremetal Kubernetes cluster using magnum. I got to the point of having the baremetal servers turn on and initiate the pxe request, however, the dnsmasq server does not respond for BOOTP requests. What are the required steps for this to work?

Update:
Not sure who closed this question. If there is something off topic please let me know.

Magnum was able to initiate the pxe requests after setting the fixed-network to the baremetal provisioning network. However, the external-network flag is also required and there is only one vlan in this setup (which has been setup as the bm-provision network). I attempted to create a public network like this (without any underlying physical network devices):

openstack network create public --provider-network-type vxlan \
                                  --external \
                                  --project service

openstack subnet create public-subnet --network public \
                                  --subnet-range 172.16.10.0/24 \
                                  --gateway 172.16.10.1 \
                                  --ip-version 4 

openstack coe cluster template create bmt \
--image fa27 \
--keypair mykey \
--external-network public \
--fixed-network bm-provision \
--fixed-subnet bm-provision-subnet \
--master-flavor bm.dev \
--flavor bm.dev \
--network-driver calico \
--coe kubernetes

openstack coe cluster create bm \
                        --cluster-template bmt \
                        --master-count 1 \
                        --node-count 1 \
                        --keypair mykey 

And this provisions the baremetal with the os over PXE via ironic, however, run into this:

{
  "default-master": "Resource CREATE failed: NotFound: resources.kube_masters.resources[0].resources.kube_master_floating: External network fa5174bb-d01d-48ca-a564-bbff283b1141 is not reachable from subnet 3e68266a-e28b-4d2c-8cd6-4042ac5a38ac.  Therefore, cannot associate Port 4950a517-31c6-4ed2-b7f8-03c3286063b3 with a Floating IP.\nNeutron server returns request_ids: ['req-bc587047-bf37-49ba-a46c-4d299f85812b']",
  "default-worker": "Resource CREATE failed: NotFound: resources.kube_masters.resources[0].resources.kube_master_floating: External network fa5174bb-d01d-48ca-a564-bbff283b1141 is not reachable from subnet 3e68266a-e28b-4d2c-8cd6-4042ac5a38ac.  Therefore, cannot associate Port 4950a517-31c6-4ed2-b7f8-03c3286063b3 with a Floating IP.\nNeutron server returns request_ids: ['req-bc587047-bf37-49ba-a46c-4d299f85812b']"
}

For this experiment, I don't need the external network, the bm-provision has a gateway pointing to nat so there is already access to the internet. Trying to achieve this with one VLAN. Will this be possible?

If you need any additional information, please let me know.

Answer
Cannot make a separate answer since this question is closed but here is what was essentially executed:

# add some more interfaces facing into the same vlan
# note macvlan was attempted also but BOOTP requests did no go through for some reason
ip link add kolla_i type veth peer name kolla_b
for i in `seq 1 10`; do ip link add p${i}_i type veth peer name p${i}_b; done
ip link add eno2_br type bridge
ip link set eno2_br up
ip link set eno2 master eno2_br
ip link set kolla_b master eno2_br
ip link set kolla_b up
ip link set kolla_i up 
ip a add 10.0.0.4/16 dev kolla_i
for i in `seq 1 10`; do ip link set p${i}_b master eno2_br; done
for i in `seq 1 10`; do ip link set p${i}_b up; done
for i in `seq 1 10`; do ip link set p${i}_i up; done

in globals.yml file (for kolla-ansible provisioning):

kolla_internal_vip_address: "10.0.0.4"
network_interface: "kolla_i"
neutron_external_interface: "p1_i,p2_i,p3_i,p4_i,p5_i,p6_i,p7_i,p8_i,p9_i,p10_i"
# this option does not exist so just add it into globals.yml
neutron_bridge_name: "br-ex1,br-ex2,br-ex3,br-ex4,br-ex5,br-ex6,br-ex7,br-ex8,br-ex9,br-ex10"
ironic_dnsmasq_interface: "p1_i"
ironic_dnsmasq_dhcp_range: "10.0.2.1,10.0.2.5"

This basically gives you 10 physical network to use within the same vlan. Here is how the ovs conf file looked after installation:

docker exec -it --user root neutron_openvswitch_agent bash -c "cat /etc/neutron/plugins/ml2/openvswitch_agent.ini" 

[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

[ovs]
bridge_mappings = physnet1:br-ex1,physnet2:br-ex2,physnet3:br-ex3,physnet4:br-ex4,physnet5:br-ex5,physnet6:br-ex6,physnet7:br-ex7,physnet8:br-ex8,physnet9:br-ex9,physnet10:br-ex10
datapath_type = system
ovsdb_connection = tcp:127.0.0.1:6640
local_ip = 10.0.0.4

The next step was to create the networks:

openstack network create \
--share \
--provider-network-type flat \
--provider-physical-network physnet1 \
--external \
provision

openstack subnet create \
--network provision \
--allocation-pool start=10.0.2.6,end=10.0.2.230 \
--gateway 10.0.0.10 \
--subnet-range 10.0.0.0/16 \
provision-subnet 


openstack network create \
--share \
--provider-network-type flat \
--provider-physical-network physnet2 \
--external \
public

openstack subnet create \
--network public \
--allocation-pool start=10.1.0.1,end=10.1.0.10 \
--allocation-pool start=10.1.0.12,end=10.1.0.250 \
--gateway 10.1.0.11 \
--subnet-range 10.1.0.0/16 \
public-subnet 

create the router

openstack router create provision-public
openstack router set provision-public --external-gateway public
openstack router add subnet provision-public provision-subnet

register the bare-metal nodes:

openstack flavor create --ram 1048576 --disk 100 --vcpus 64 bm.dev
openstack flavor set --property baremetal=true bm.dev
openstack flavor set --property resources:CUSTOM_BAREMETAL_DEV=1 bm.dev
openstack flavor set --property resources:VCPU=0 bm.dev
openstack flavor set --property resources:MEMORY_MB=0 bm.dev
openstack flavor set --property resources:DISK_GB=0 bm.dev


openstack baremetal node create --name dev02 \
--driver ipmi \
--driver-info ipmi_username=<user> \
--driver-info ipmi_password=<pass> \
--driver-info ipmi_address=<ipmi_addr> \
--driver-info deploy_kernel=http://10.0.0.4:8089/ironic-agent.kernel \
--driver-info deploy_ramdisk=http://10.0.0.4:8089/ironic-agent.initramfs \
--driver-info cleaning_network=provision \
--driver-info provisioning_network=provision \
--deploy-interface=direct \
--network-interface=flat \
--driver-info force_persistent_boot_device=True \
--property capabilities=boot_mode:uefi \
--property cpu_arch=x86_64 \
--property local_gb=1000 \
--resource-class baremetal.dev

openstack baremetal port create <mac_addr> --node <id>
openstack baremetal node manage dev02
openstack baremetal node provide dev02

openstack baremetal node create --name dev03 \
--driver ipmi \
--driver-info ipmi_username=<user> \
--driver-info ipmi_password=<pass> \
--driver-info ipmi_address=<ipmi_addr> \
--driver-info deploy_kernel=http://10.0.0.4:8089/ironic-agent.kernel \
--driver-info deploy_ramdisk=http://10.0.0.4:8089/ironic-agent.initramfs \
--driver-info cleaning_network=provision \
--driver-info provisioning_network=provision \
--deploy-interface=direct \
--network-interface=flat \
--driver-info force_persistent_boot_device=True \
--property capabilities=boot_mode:uefi \
--property cpu_arch=x86_64 \
--property local_gb=1000 \
--resource-class baremetal.dev

openstack baremetal port create <mac_addr> --node <id>
openstack baremetal node manage dev03 
openstack baremetal node provide dev03

docker exec --user root  nova_conductor bash -c "nova-manage cell_v2 discover_hosts --by-service"

create template and deploy

openstack coe cluster template create bmt \
--image bm \
--keypair mykey \
--external-network public \
--fixed-network provision \
--fixed-subnet provision-subnet \
--master-flavor bm.dev \
--flavor bm.dev \
--network-driver calico \
--coe kubernetes

openstack coe cluster create bm \
                        --cluster-template bmt \
                        --master-count 1 \
                        --node-count 1 \
                        --keypair mykey

Note, this is not for production since all networks will be in the same broadcast domain but suitable for experiments.

Any attempt to start nova compute services fails with the error HypervisorUnavailable: Connection to the hypervisor is broken on host:

The trace log is the following (taken from /var/log/nova/nova-compute.log )

AUDIT nova.service [-] Starting compute node (version 2014.1.1)
ERROR nova.virt.libvirt.driver [-] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
TRACE nova.virt.libvirt.driver Traceback (most recent call last):
TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 787, in _connect
TRACE nova.virt.libvirt.driver     libvirt.openAuth, uri, auth, flags)
TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
TRACE nova.virt.libvirt.driver     rv = execute(f,*args,**kwargs)
TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
TRACE nova.virt.libvirt.driver     rv = meth(*args,**kwargs)
TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
TRACE nova.virt.libvirt.driver     if ret is None:raise libvirtError('virConnectOpenAuth() failed')
TRACE nova.virt.libvirt.driver libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
TRACE nova.virt.libvirt.driver
ERROR nova.openstack.common.threadgroup [-] Connection to the hypervisor is broken on host: <HOST_NAME>
TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 117, in wait
TRACE nova.openstack.common.threadgroup     x.wait()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 49, in wait
TRACE nova.openstack.common.threadgroup     return self.thread.wait()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 483, in run_service
TRACE nova.openstack.common.threadgroup     service.start()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 163, in start
TRACE nova.openstack.common.threadgroup     self.manager.init_host()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1006, in init_host
TRACE nova.openstack.common.threadgroup     self.driver.init_host(host=self.host)
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 656, in init_host
TRACE nova.openstack.common.threadgroup     self._do_quality_warnings()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 639, in _do_quality_warnings
 TRACE nova.openstack.common.threadgroup     caps = self.get_host_capabilities()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2849, in get_host_capabilities
TRACE nova.openstack.common.threadgroup     xmlstr = self._conn.getCapabilities()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 721, in _get_connection
TRACE nova.openstack.common.threadgroup     wrapped_conn = self._get_new_connection()
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 674, in _get_new_connection
TRACE nova.openstack.common.threadgroup     wrapped_conn = self._connect(self.uri(), self.read_only)
TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 796, in _connect
TRACE nova.openstack.common.threadgroup     raise exception.HypervisorUnavailable(host=CONF.host)
TRACE nova.openstack.common.threadgroup HypervisorUnavailable: Connection to the hypervisor is broken on host: <HOST_NAME>

No other log file indicates any error syslog, dmesg and /var/log/libvirt/libvirtd.log does not show any issue

The libvirt configuration (/etc/libvirt/libvirtd.conf) was set to log the errors log_outputs="3:syslog:libvirtd"

Before the error appear the node was running smoothly, no changes were made.