GPU PASSTHROUGH GUIDE FOR UBUNTU 22.04 LTS

This guide was tested on Ubuntu 22.04 LTS with Nouveau driver and the following system configuration:

• MSI® motherboard Z170A
• Intel® Core™ i5-6600 Processor
• GeForce GTX 1650 AERO ITX 4G OC.

Attention! This guide is only relevant for Nouveau driver. Please read the text carefully before you start system changes.

HARDWARE SPECIFICATION REQUIRED FOR THIS GUIDE

For Intel based system:

• Motherboard with Integrated Graphics (for manipulations)

• CPU with VT-x, VT-d and Processor Graphics

• Discrete Graphics Card with GPU passthrough support (for passthrough)

• Monitor with two inputs like Displayport, D-SUB, DVI, HDMI or switcher box with monitor or two monitors.

For AMD based system:

• Motherboard with Integrated Graphics (for manipulations)

• CPU with AMD-V, AMD-Vi and Integrated Graphics

• Discrete Graphics Card with GPU passthrough support (for passthrough)

• Monitor with two inputs like Displayport, D-SUB, DVI, HDMI or switcher box with monitor or two monitors.

See list of IOMMU-supporting hardware (list may not be complete).

Also You will need 8GB RAM for operating system (OS) and 8GB RAM for guest machine, the 16GB RAM may significantly increase your HDD's or SSD's performance and lifetime.

REQUIRED STEPS

STEP 1. Enable Hardware-assisted virtualization in the BIOS

• For Intel based system enable Intel Virtualization Tech (VT-x) and Intel VT-D Tech

• For AMD based system enable SVM Mode.

STEP 2. Set the BIOS to use the Integrated Graphics as the primary boot device

Attention! If you boot the system with designed for passthrough PCI-Express Graphics Device, error code 43* will appear.

STEP 3. Check for IOMMU Support on your CPU

For AMD processor:

$ cat /proc/cpuinfo | grep --color svm

For Intel processor:

$ cat /proc/cpuinfo | grep --color vmx

You should see the highlighted text svm or vmx.

STEP 4. Enable IOMMU

$ sudo nano /etc/default/grub

Add the following options to GRUB_CMDLINE_LINUX_DEFAULT=""

For AMD processor:

amd_iommu=on kvm.ignore_msrs=1

For Intel processor:

intel_iommu=on kvm.ignore_msrs=1

STEP 5. Update GRUB

$ sudo grub-mkconfig -o /boot/grub/grub.cfg

STEP 6. Reboot your system

STEP 7. Check that IOMMU is enabled

$ sudo dmesg | grep -i -e DMAR -e IOMMU

You should see message like below:

[    0.123456] pci 0000:09:00.0: Adding to iommu group 1
[    0.234567] pci 0000:0a:00.0: Adding to iommu group 2
[    0.345678] DMAR: Intel(R) Virtualization Technology for Directed I/O

STEP 8. Find your device

$ lspci -nnk

You should see text like below:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10aa:10bb] (rev a1) (prog-if 00 [VGA controller])
    Subsystem: Micro-Star International Co., Ltd. [MSI] TU117 [GeForce GTX 1650] [0101:a1a1]
    Flags: bus master, fast devsel, latency 0, IRQ 151, IOMMU group 1
    Memory at de000000 (32-bit, non-prefetchable) [size=16M]
    Memory at c0000000 (64-bit, prefetchable) [size=256M]
    Memory at d0000000 (64-bit, prefetchable) [size=32M]
    I/O ports at e000 [size=128]
    Expansion ROM at 000c0000 [disabled] [size=128K]
    Capabilities: <access denied>
    Kernel driver in use: nouveau
    Kernel modules: nvidiafb, nouveau

01:00.1 Audio device [0403]: NVIDIA Corporation Device [01cc:01ee] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [0202:a2a2]
    Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 1
    Memory at df080000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: <access denied>
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel

If You see text "Kernel driver in use: nvidia" like below:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10aa:10bb] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] TU117 [GeForce GTX 1650] [0101:a1a1]
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation Device [01cc:01ee] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [0202:a2a2]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel

your system uses NVIDIA Proprietary Drivers. You need to remove all old video drivers. Here is a recipe which removes all old video drivers, and reinstalls nouveau:

$ sudo nvidia-settings --uninstall
$ sudo apt-get remove --purge nvidia*
$ sudo apt-get remove --purge xserver-xorg-video-nouveau
$ sudo apt-get remove --purge xserver-xorg-video-nv
$ sudo apt-get install nvidia-common
$ sudo apt-get install xserver-xorg-video-nouveau
$ sudo apt-get install xserver-xorg-video-all
$ sudo apt-get install --reinstall libgl1-mesa-glx libgl1-mesa-dri
$ sudo apt-get install --reinstall xserver-xorg-core
$ sudo dpkg-reconfigure xserver-xorg

Run autoremove:

$ sudo apt autoremove

Then reboot your system and repeat the STEP 8.

STEP 9. Create a new file called vfio.conf

$ sudo nano /etc/modprobe.d/vfio.conf

Add the following lines with your device IDs from the STEP 8:

blacklist nouveau
blacklist snd_hda_intel
options vfio-pci ids=10aa:10bb,01cc:01ee

STEP 10. Update the existing initramfs

$ sudo update-initramfs -u

STEP 11. Reboot your system

STEP 12. Make sure everything is OK

$ lspci -nnk

You should see text like below:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10aa:10bb] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] TU117 [GeForce GTX 1650] [0101:a1a1]
    Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation Device [01cc:01ee] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [0202:a2a2]
    Kernel modules: snd_hda_intel

STEP 13. Use VirtManager to create a new guest machine

Install KVM

$ sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils

Add user

$ sudo adduser username libvirtd

If you have group issue "adduser: The group `libvirtd' does not exist." try:

$ sudo addgroup libvirtd

Install VirtManager

$ sudo apt-get install virt-manager

Create new guest machine with a custom configuration.

$ virt-manager

You must:

a) select the "Q35" chipset and "UEFI x86_64:/usr/share/OVMF/OVMF_CODE_4M.ms.fd" firmware or similar

b) add PCI Host Devices** for GPU passthrough

c) set QXL video driver and SPICE client for display with "Address" listen type.

STEP 14. Edit the guest machine

$ virsh list --all

$ sudo virsh edit YourGuestMachineName

Add the following lines:

<vendor_id state='on' value='1234567890ab'/>
<kvm>
 <hidden state='on'/>
</kvm>
<ioapic driver='kvm'/>

The end result should look something like:

<features>
  <acpi/>
  <apic/>
  <hyperv>
    <relaxed state='on'/>
    <vapic state='on'/>
    <spinlocks state='on' retries='8191'/>
    <vendor_id state='on' value='1234567890ab'/>
  </hyperv>
  <kvm>
    <hidden state='on'/>
  </kvm>
  <vmport state='off'/>
  <ioapic driver='kvm'/>
</features>

Once your done editing, use CTRL+x to save the changes.

STEP 15. Run the guest machine

NOTE

If you want to use the PCI-Express Graphics Device as the primary boot device, do the following:

$ sudo nano /etc/modprobe.d/vfio.conf

Change the following lines with your device IDs from the STEP 8:

blacklist nouveau
blacklist snd_hda_intel
options vfio-pci ids=10aa:10bb,01cc:01ee

with:

#blacklist nouveau
#blacklist snd_hda_intel
#options vfio-pci ids=10aa:10bb,01cc:01ee

Update the existing initramfs

$ sudo update-initramfs -u

Reboot your system and set the BIOS to use the PCI-Express Graphics Device as the primary boot device.

TROUBLESHOOTING

Problem: the guest machine start normally but I see black screen.

Action:

• check the cable connections and the settings of your monitor

• check the steps 1, 2, 14

• install the correct device driver on Windows

• uninstall and reinstall the device driver on Windows

• update the device driver on Windows

REFERENCES

• KVM GPU Passthrough Tutorial

• Fighting error 43 – Nvidia GPU in a virtual machine

• Fix graphics device problems with error code 43

• VideoDriverDetection

• YouTube | GPU passthrough guide for Ubuntu 20.04

* Windows has stopped this device because it has reported problems (code 43).

** For example the NVIDIA Corporation TU117 [GeForce GTX 1650] and NVIDIA Corporation devices