Alex Vrublevskiy's questions

We've faced weird issue with one of volumeMounts of one of our deployments. Here are relevant manifests' snippets:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: xxx-media-pv-resource-claim
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: cephfs
...
apiVersion: extensions/v1beta1
kind: Deployment
...
        volumeMounts:
        - mountPath: /opt/media/logs
          name: xxx-media-log-storage
        - mountPath: /shared-media-files
          name: xxx-media-resource-storage
...
      volumes:
      - name: xxx-media-log-storage
        persistentVolumeClaim:
          claimName: xxx-media-pv-log-claim
      - name: xxx-media-resource-storage
        persistentVolumeClaim:
          claimName: xxx-media-pv-resource-claim
...

We get read/write error: "Operation not permitted" when trying to, e.g. cat/echo from/to files in /shared-media-files although: a) can successfully touch/mkdir/chmod/rm there; b) can do everything in /opt/media/logs which is backed up by identical PVC (except name). Cephfs storageclass and corresponding Ceph cluster is a storage for whole k8s cluster and all k8s entities that use it except xxx-media-resource-storage behave as expected.

k8s master: v1.14.1
k8s workers: v1.16.0
cephfs-provisioner: v2.1.0-k8s1.11
Ceph: 14.2.1

We checked everything obvious like Ceph cluster health, free storage space, file permissions and ACLs, mount mode, PV/PVC descriptions, logs and didn't find anything special. As xxx-media-pv-resource-claim PVC accessMode is RWX, we tried to use it with another test pod and got same errors. We tried to google the issue as well but with no luck.

Please help to troubleshoot the issue.

We're trying to deploy OpenStack on hardened Debian-based Linux distro via Kolla-ansible and we seem to be almost done but facing the issue with the nova_compute container which complaints:

2021-09-12 08:56:34.365 7 INFO nova.virt.libvirt.driver [-] Connection event '0' reason. 
 'Failed to connect to libvirt: Unable to query peer security context: No data available

and restarts permanently.

Here are relevant config snippets:

(venv) root@server11:~# grep nova /etc/kolla/globals.yml 
nova_backend_ceph                  : "yes"
nova_compute_virt_type             : "qemu"
(venv) root@server11:~# cat /etc/kolla/config/nova/nova-compute.conf 
[libvirt]
virt_type=qemu
cpu_mode = none
(venv) root@server11:~# 

Tried to figure out the root cause myself, read the docs, googled but to no avail, so any suggestions on how to fix the issue will be highly appreciated.