Home / server / Questions / 1057346
In Process
BakaKuna
Asked: 2021-03-18 02:02:50 +0800 CST

What is the best way to repair a situation where the nova database contains an attachment, but cinder doesn't?

  • 772

In our OpenStack cloud there are a server with server_id and a volume with <volume_id>. We are in a situation where there is a volume that appears to be attached from the perspective of Nova, but not from the perspective of Cinder. There is a record in the nova database for this attachment: MariaDB [(none)]> select * from block_device_mapping where instance_uuid = "<volume_id>", shows the entry.

Nova shows the attachment:

openstack server show <server_id>
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+---------------------------------------------------------
| volumes_attached                    | id='...'                                                 |
|                                     | id='<volume_id'                                          |
+-------------------------------------+----------------------------------------------------------+

Cinder, does not know about this attachment:

openstack volume show <volume_id>
+--------------------------------+--------------------------------------+
| Field                          | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |

From within the VM, the volume seems connected and can be operated on.

I tried setting the volume to an available and detached state, but it is not possible to attach it in any state that I have tried.

It also tried removing the attachment completely. When running openstack server remove volume or nova volume-detach the command finishes without an error and the cinder logging shows: Roll detaching of volume completed successfully. However, the situation does not change.

What would be the best way to recover from this situation? For this specific VM, I think I could remove the database entry from block_device_mapping and rebuild the VM? For the future I would like to know if it is also possible to repair the situation in OpenStack to reflect the situation in the real world. That means, is there any way in which I can add the attachment, so that it is also visible in Cinder?

openstack cinder openstack-nova

1 Answers

  1. For any case if someone if still looking for the solution: You need to find record in table block_device_mapping (nova DB) and delete it.

    select * from nova.block_device_mapping where deleted_at is NULL and volume_id = 'vol_id'\G

    Delete it carefully with transaction, because wrong action could bring a lot of harm:

    BEGIN;
UPDATE nova.block_device_mapping SET deleted=id, deleted_at=NOW() WHERE id='vol_id' LIMIT 1;
#select to check that you have made it right and, if all is correct, COMMIT (ROLLBACK if it is wrong)
COMMIT;

    After that hard reboot the VM.

    In cinder DB you can check volume_attachment table (to be sure)

    select * from cinder.volume_attachment where volume_id = '' and deleted <> '1'\G
    • 1