PCI Passthrough

Many guide on the internet seem outdated, on my test it didn't work.

In this guide, how to PCI Passthrough on Proxmox version 6.4-4 or above

Update Proxmox

After you install, it's good to check for update without subscription.

Optional

Add unstable source

nano /etc/apt/sources.list

Add new line

# non production use
deb http://download.proxmox.com/debian buster pve-no-subscription

Required

Remove subscription source

nano /etc/apt/sources.list.d/pve-enterprise.list

Add # at the begining of line 1, it should look like this:

#deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

Update & Upgrade

apt update
apt dist-upgrade

Enable IOMMU

Before proceed, make sure check that your processor and motherboard has iommu support, refer to motherboard manufacturer.

For Ryzen system, navigate to AMD CBS\NBIO Common Options, make sure enable IOMMU and ACS Enable

Edit GRUB

nano /etc/default/grub

Add amd_iommu=on or intel_iommu=on on GRUB_CMDLINE_LINUX_DEFAULT after quiet

It should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

Save it, and run this command to update GRUB

update-grub

VFIO Modules

For Proxmox 6.4, VFIO Module already enabled, you can skip this step, but adding it no harm do, just to be safe:

Edit

nano /etc/modules

Add & Save

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Reduce lag and interrupts

Prevent some odd issue and sound cracking problem. this may required!

Paste this line on the terminal:

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Blacklist Driver

Prevent driver from loading that prevent VFIO to load.

Paste this line on the terminal:

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidiafb" >> /etc/modprobe.d/blacklist.conf
echo "blacklist snd_hda_intel" >> /etc/modprobe.d/blacklist.conf

Add Device

Now add PCI Device that want to passthrough.

Find

Now, find device that you want, enter this command on the terminal

lspci

Results should be look like this:

0b:00.0 VGA compatible controller: NVIDIA Corporation Device 2206 (rev a1)
0b:00.1 Audio device: NVIDIA Corporation Device 1aef (rev a1)
0c:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)
0c:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)

Device number 0b:00.0 and 0c:00.0 is my GPU, ommit .0 leaving like this 0b:00 and 0c:00

Now we going to find Device ID, simply type:

lspci -n -s 0b:00

Results should look like his:

0b:00.0 0300: 10de:2206 (rev a1)
0b:00.1 0403: 10de:1aef (rev a1)

Repeat same for 2nd GPU:

lapci -n -s 0c:00

0c:00.0 0300: 10de:1c03 (rev a1)
0c:00.1 0403: 10de:10f1 (rev a1)

Now, copy Device ID: 10de:2206, 10de:1aef, 10de:1c03 and 10de:10f1

Update /etc/modprobe.d/vfio.conf

Compile Device ID in one line with comma:

10de:2206,10de:1aef,10de:1c03,10de:10f1

Paste and should look like this:

echo "options vfio-pci ids=10de:2206,10de:1aef,10de:1c03,10de:10f1 disable_vga=1" > /etc/modprobe.d/vfio.conf
update-initramfs -u

Check vfio

After your server booted, run lspci -knn, find your GPU and make sure vfio driver is loaded!