Questions[openstack](server)

I don't know how to troubleshoot this any further. I hope I can find help.

When I delete a compute service with: openstack compute service delete a04910ef-1441-4949-8ffb-6393c22141b2, I can restart the nova-compute on the machine with sudo systemctl restart nova-compute to get back the service in openstack.

However, the node is not listed anymore in openstack hypervisor list (os-compute03.maas for instance in my case):

$ openstack compute service list --service nova-compute
+--------------------------------------+--------------+-------------------+------+----------+-------+----------------------------+
| ID                                   | Binary       | Host              | Zone | Status   | State | Updated At                 |
+--------------------------------------+--------------+-------------------+------+----------+-------+----------------------------+
| d1fadd40-6035-4f76-b8c1-5b981d003832 | nova-compute | os-compute08.maas | nova | disabled | down  | 2024-09-25T14:02:55.000000 |
| 7e02a0bd-0e53-45bc-9680-99a33d98c05b | nova-compute | os-compute04.maas | nova | enabled  | up    | 2024-10-18T11:43:12.000000 |
| 7f47d65e-b041-44dc-927e-085effdf0ec9 | nova-compute | os-compute09.maas | nova | enabled  | up    | 2024-10-18T11:43:07.000000 |
| 3a7527b3-3664-4ae1-ac90-ac04e835ee5b | nova-compute | os-compute03.maas | nova | enabled  | up    | 2024-10-18T11:43:13.000000 |
+--------------------------------------+--------------+-------------------+------+----------+-------+----------------------------+

$ openstack hypervisor list
+----+---------------------+-----------------+----------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP  | State |
+----+---------------------+-----------------+----------+-------+
|  1 | os-compute08.maas   | QEMU            | 10.0.1.8 | down  |
|  2 | os-compute04.maas   | QEMU            | 10.0.1.4 | up    |
|  3 | os-compute09.maas   | QEMU            | 10.0.1.9 | up    |
+----+---------------------+-----------------+----------+-------+

^ no os-compute03.maas

$ sudo nova-manage cell_v2 list_hosts
                                                                                                                                     
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
+-----------+--------------------------------------+-------------------+
| Cell Name |              Cell UUID               |      Hostname     |
+-----------+--------------------------------------+-------------------+
|   cell1   | d3dfd353-b4ee-4293-b362-1e0175ebe337 | os-compute04.maas |
|   cell1   | d3dfd353-b4ee-4293-b362-1e0175ebe337 | os-compute08.maas |
|   cell1   | d3dfd353-b4ee-4293-b362-1e0175ebe337 | os-compute09.maas |
+-----------+--------------------------------------+-------------------+

$ sudo nova-manage cell_v2 discover_hosts --verbose
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
Found 2 cell mappings.
Skipping cell0 since it does not contain hosts.
Getting computes from cell 'cell1': d3dfd353-b4ee-4293-b362-1e0175ebe337
Found 0 unmapped computes in cell: d3dfd353-b4ee-4293-b362-1e0175ebe337

I tried equally to restart nova controller:

$ sudo systemctl restart nova-scheduler
$ sudo systemctl restart nova-conductor

When restarting the nova controller, the logs does say the following:

2024-10-18 11:53:44.939 630048 INFO nova.scheduler.host_manager [None req-eb8c8117-136c-4a13-ac57-d32d4099388a - - - - - -] Received a sync request from an unknown host 'os-compute09.maas'. Re-created its InstanceList.
2024-10-18 11:53:44.939 630046 INFO nova.scheduler.host_manager [None req-eb8c8117-136c-4a13-ac57-d32d4099388a - - - - - -] Received a sync request from an unknown host 'os-compute09.maas'. Re-created its InstanceList.
2024-10-18 11:53:44.940 630045 INFO nova.scheduler.host_manager [None req-eb8c8117-136c-4a13-ac57-d32d4099388a - - - - - -] Received a sync request from an unknown host 'os-compute09.maas'. Re-created its InstanceList.
2024-10-18 11:53:44.968 630047 INFO nova.scheduler.host_manager [None req-eb8c8117-136c-4a13-ac57-d32d4099388a - - - - - -] Received a sync request from an unknown host 'os-compute09.maas'. Re-created its InstanceList.
2024-10-18 11:54:02.539 630048 INFO nova.scheduler.host_manager [None req-ad410200-f99f-4dc7-896d-637dd9620568 - - - - - -] Received a sync request from an unknown host 'os-compute04.maas'. Re-created its InstanceList.
2024-10-18 11:54:02.541 630047 INFO nova.scheduler.host_manager [None req-ad410200-f99f-4dc7-896d-637dd9620568 - - - - - -] Received a sync request from an unknown host 'os-compute04.maas'. Re-created its InstanceList.
2024-10-18 11:54:02.542 630046 INFO nova.scheduler.host_manager [None req-ad410200-f99f-4dc7-896d-637dd9620568 - - - - - -] Received a sync request from an unknown host 'os-compute04.maas'. Re-created its InstanceList.
2024-10-18 11:54:02.543 630045 INFO nova.scheduler.host_manager [None req-ad410200-f99f-4dc7-896d-637dd9620568 - - - - - -] Received a sync request from an unknown host 'os-compute04.maas'. Re-created its InstanceList.
2024-10-18 11:55:06.432 630048 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Host mapping not found for host os-compute03.maas. Not tracking instance info for this host.
2024-10-18 11:55:06.432 630048 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Received a sync request from an unknown host 'os-compute03.maas'. Re-created its InstanceList.
2024-10-18 11:55:06.435 630047 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Host mapping not found for host os-compute03.maas. Not tracking instance info for this host.
2024-10-18 11:55:06.435 630046 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Host mapping not found for host os-compute03.maas. Not tracking instance info for this host.
2024-10-18 11:55:06.436 630045 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Host mapping not found for host os-compute03.maas. Not tracking instance info for this host.
2024-10-18 11:55:06.436 630047 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Received a sync request from an unknown host 'os-compute03.maas'. Re-created its InstanceList.
2024-10-18 11:55:06.437 630045 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Received a sync request from an unknown host 'os-compute03.maas'. Re-created its InstanceList.
2024-10-18 11:55:06.436 630046 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Received a sync request from an unknown host 'os-compute03.maas'. Re-created its InstanceList.

And particularly: Host mapping not found for host os-compute03.maas. Not tracking instance info for this host. I guess this is when the node should be added in the nova cell_v2? And this is indeed not the case.

I guess the following records in the database concerns the compute-nodes:

mysql> SELECT hypervisor_hostname, host_ip FROM nova.compute_nodes;
+---------------------+----------+
| hypervisor_hostname | host_ip  |
+---------------------+----------+
| os-compute08.maas   | 10.0.1.8 |
| os-compute04.maas   | 10.0.1.4 |
| os-compute09.maas   | 10.0.1.9 |
| os-compute03.maas   | 10.0.1.3 |
+---------------------+----------+
4 rows in set (0.00 sec)

os-compute03.maas is indeed not listed in the nova_api.host_mappings table:

SELECT * FROM nova_api.host_mappings;
+---------------------+------------+----+---------+-------------------+
| created_at          | updated_at | id | cell_id | host              |
+---------------------+------------+----+---------+-------------------+
| 2024-07-02 21:23:10 | NULL       |  1 |       2 | os-compute08.maas |
| 2024-07-02 21:23:11 | NULL       |  2 |       2 | os-compute04.maas |
| 2024-07-02 21:23:11 | NULL       |  3 |       2 | os-compute09.maas |
+---------------------+------------+----+---------+-------------------+

I did not tried to add the host manually in this table. I suppose this should be done automatically, but when? What did I miss?

How can we further debug this situation?

I have an Openstack (yoga) cluster that was installed using kolla-ansible version 14.1.0.

I'm trying to set up OpenIDConnect to login through the Horizon interface and I therefore followed this documentation.

The problem I'm experiencing is that when trying to connect via the new identity provider I set up, I get sent to this URL, which responds 404 :

http://<openstack_hostname>:5000/auth/OS-FEDERATION/identity_providers/<idp_id>/protocols/openid/websso?origin=http://<openstack_hostname>/auth/websso/

From what I can gather there is a /v3/ missing at the beginning of the path, and it should instead be :

http://<openstack_hostname>:5000/v3/auth/OS-FEDERATION/identity_providers/<idp_id>/protocols/openid/websso?origin=http://<openstack_hostname>/auth/websso/

And when I manually enter that url it successfully sends me to the identity provider to continue the login process.

I do not know where this is coming from. As far as I can tell by looking up the documentation the URL should be with a /v3/ but I do not get where this wrong URL comes from in my installation.

I'm quite new to all this *aaS thing, and currently I'm trying to wrap my head around concepts and how things basically work.

What puzzles me quite a bit at the moment is the difference between a variety of products, especially the following three:

• OpenStack
• CloudFoundry
• Stackato

The last one is the easiest for me to grasp: It's basically a software that manages application and service instances so you can push a custom-developed application to it and tell it to run this on a number of instances, so Stackato takes care of how to distribute the application.

But now I've read that Stackato is related to CloudFoundry (see http://strongloop.com/strongblog/in-the-loop-stackato-a-platform-as-a-service-that-you-can-deploy-and-manage-yourself/). Are they basically two products similar to each other that you can exchange for one another? Is the one a fork of the other (like Jenkins and Hudson), or how are they related?

And then I've read that you can Install CloudFoundry on top of OpenStack. I initially had thought that OpenStack is basically just another option, but apparently it's not, but something more low-level.

Can anybody please shed some light on this, and give me some hints?

In setting up our OpenStack environment, I ran into a problem that was preventing instances from contacting a server running on the host. The metadata service (which exposes an HTTP API) runs on port 8775 on the host, and the OpenStack networking code adds the following DNAT rule to grant access via a special address on port 80:

-A PREROUTING -d 169.254.169.254/32 
  -p tcp -m tcp --dport 80 -j DNAT --to-destination 127.0.0.1:8775 

Instances are connected to the host via a local bridge device, and 169.254.169.254 is assigned to lo.

While this rule successfully matches packets originating from a guest instance trying to access http://169.254.168.254/, they never reach the listening service.

Replacing this DNAT rule with a largely equivalent REDIRECT makes everything work correctly:

-A PREROUTING -d 169.254.169.254/32 
  -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8775 

I'm trying to understand why the REDIRECT works and the DNAT fails. It's not clear from the iptables documentation whether or not the DNAT rule is expected to work for locally destined traffic. I'm hoping someone here can provide an authoritative answer, ideally backed up by documentation, or can suggest what might be amiss in my configuration that is preventing the DNAT rule from working as expected.

I'm testing Openstack and i'd like to know how can i find out about resource utilization inside instances (Virtual Machines).

Openstack uses kvm, qemu and libvirt in my testing environment. Is there any way to get those stats without logging in machines (CPU, Memory, Swap, Network traffic)? I'd like collect those data from hypervisors or through libvirt.

Any suggestions?