Skip to content

Instantly share code, notes, and snippets.

@vadimstasiev

vadimstasiev/.readme.md

Created July 17, 2022 22:08
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save vadimstasiev/f54c7342e4a4ec779115798665db1a6c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save vadimstasiev/f54c7342e4a4ec779115798665db1a6c to your computer and use it in GitHub Desktop.
Download ZIP
Setting Up Google Coral In Docker within an LXC container
Raw
.readme.md

Copy paste of Reddit post, for archival purpose, original post: https://www.reddit.com/r/Proxmox/comments/n34f8q/proxmox_vm_ubuntu_2004_frigate_2x_google_coral_tpu/

Proxmox >> VM (Ubuntu 20.04) >> Frigate + 2x Google Coral TPU This assumes that you already have Proxmox (6.3+) installed and a VM running Ubuntu 20.04+ with Docker installed and a Frigate container created that you wish to pass-thru some Google Coral(s) for TensorFlow processing.

Required Hardware

You will need a Coral device. These can be purchased from okdo.com (there are other distributors)

Note: I avoided the USB device due to the high price and the low availably so instead opted to try two different PCIe devices. I can confirm both devices below work but if I had to choose I would go the mini-pcie route as it's a smaller form factor with equal performance of the m.2

Model Shop URL Coral URL Price (inc. VAT) Coral Mini PCIe M.2 Accelerator B/M Key https://www.okdo.com/p/coral-mini-pcie-m-2-accelerator-b-m-key/ https://coral.ai/products/m2-accelerator-bm £23.40 Coral Mini PCIe Accelerator https://www.okdo.com/p/coral-mini-pcie-accelerator/ https://coral.ai/products/pcie-accelerator £23.40 The motherboard that I am using does not have an m.2 nor a Mini PCIe shot so I also needed an adapter.

Shop URL Use with Price (Inc. VAT) M.2 to PCI-E 3.0 x4 Adapter https://www.amazon.co.uk/ Coral Mini PCIe M.2 Accelerator B/M Key £6.29 Mini PCI-E To PCI-E Adapter https://www.ebay.co.uk/ Coral Mini PCIe Accelerator £5.74 note: the prices listed for the adapters appear to have changed since I purchased them but identical ones can be found if you search on the same sites.

I've uploaded the 3D files for low-profile brackets for the 2 adapters shown in the above table. I've only just created my Thingiverse account so the account is on-hold for 24hours. The designs should be downloadable from 4th May 2021. ** The Thingiverse URLs are now live **

https://www.thingiverse.com/thing:4846922 ( Low Profile Bracket for MiniPCI Adapter )https://www.thingiverse.com/thing:4846916 ( Low Profile Bracket for M.2 Adapter )

Proxmox PCI passthrough to VM

Shutdown the server and Insert the card into a PCIe slot. Start it up and check for PCI device being detected

root@proxmox:~# lspci -nnk | grep 089a

04:00.0 System peripheral [0880]: Device [1ac1:089a]Subsystem: Device [1ac1:089a]

The 04.00.0 is the important part. This will change based upon your system and/or PCIe port used.

if you have 2x devices you’ll have something like:

root@proxmox:~# lspci -nnk | grep 089a

02:00.0 System peripheral [0880]: Device [1ac1:089a]Subsystem: Device [1ac1:089a] 04:00.0 System peripheral [0880]: Device [1ac1:089a]Subsystem: Device [1ac1:089a]

Make sure 04:00.0 is in it’s own iommu group

root@proxmox:~# find /sys/kernel/iommu_groups/ -type l

/sys/kernel/iommu_groups/34/devices/0000:7f:0a.1 /sys/kernel/iommu_groups/34/devices/0000:7f:0a.2 /sys/kernel/iommu_groups/6/devices/0000:00:04.0 /sys/kernel/iommu_groups/62/devices/0000:80:04.2 /sys/kernel/iommu_groups/90/devices/0000:ff:10.1 /sys/kernel/iommu_groups/24/devices/0000:04:00.0

This will show a long list but I prefer to grep for the device bus ID and then grep again the iommu group

root@proxmox:~# find /sys/kernel/iommu_groups/ -type l | grep 04:

/sys/kernel/iommu_groups/24/devices/0000:04:00.0

Then again on the group

root@proxmox:~# find /sys/kernel/iommu_groups/ -type l | grep /24/

/sys/kernel/iommu_groups/24/devices/0000:04:00.0

Repeat for the number of devices you have

Blacklist the Device (on Proxmox)

Create a new file at /etc/modprobe.d/blacklist-apex.conf and add these lines:

root@proxmox:~# nano /etc/modprobe.d/blacklist-apex.conf

blacklist gasket blacklist apex options vfio-pci ids=1ac1:089a Save the file and then then run the following to apply the changes

root@proxmox:~# update-initramfs -u -k all

The command above will take a while to run. Once completed restart ProxMox

root@proxmox:~# reboot

Verify that the apex and gasket modules did not load by running this:

root@proxmox:~# lsmod | grep apex

It should print nothing.

Add PCIe Device to VM

I am running Ubuntu 20.04 and the driver is pre-installed and works fine on Frigate but you may wish to install the latest drivers.

Once rebooted with the host machine stopped, add the PCI device to the host (where 101 is the VM ID in Proxmox)

root@proxmox:~# qm set 101 -hostpci0 04:00.0

or if you have 2x devices then pass them both through

root@proxmox:~# qm set 101 -hostpci0 04:00.0

root@proxmox:~# qm set 101 -hostpci1 02:00.0

On the VM (Not ProxMox)

Once the VM is loaded check for the PCI device

ubuntu2004:~$ lspci -nnk | grep 089a

00:10.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a] Subsystem: Global Unichip Corp. Coral Edge TPU [1ac1:089a]

Check for Apex drivers

ubuntu2004:~$ lsmod | grep apex

apex 20480 5gasket 106496 6 apex

Check for Linux Device

the directory should be present. If not, you have an issue

ubuntu2004:~$ ls -al /dev/apex_0

Note: You can install the latest drivers if you wish but I have experienced no issues so I haven't installed them. https://coral.ai/docs/m2/get-started/

Add PCI device to Frigate Compose

You need to allow the device to be used by the frigate container. Add the following into the docker-compose.yaml file:

devices:
  - /dev/apex_0:/dev/apex_0

Or if you have 2 or more devices

devices:
  - /dev/apex_0:/dev/apex_0
  - /dev/apex_1:/dev/apex_1

Remember to re-compose the container for the change to take affect

Check the device is present in the Frigate container

Once the container has been re-compose you can check the driver is loaded okay

ubuntu2004:~$ docker exec -it frigate bash

root@ee9e73933c85:/opt/frigate# ls -al /dev/apex_0

crw------- 1 root root 120, 0 Apr 21 18:41 /dev/apex_0

Success. Type exit to leave the Frigate bash prompt

root@ee9e73933c85:/opt/frigate# exit

Configure Frigate config.yml

Hash out or delete the existing CPU detectors and replace with the following:

detectors: coral_pci: type: edgetpu device: pci or if you have 2 or more:

detectors: coral_pci1: type: edgetpu device: pci:0 coral_pci2: type: edgetpu device: pci:1 note: coral_pci1 is just a label. It’s not doing anything more than giving the device a friendly name.

Save the changes and restart Frigate

ubuntu2004:~$ docker restart frigate

You can check the TPU is loaded correctly in docker logs

ubuntu2004:~$ docker log frigate

Starting migrationspeewee_migrate INFO : Starting migrations There is nothing to migrate peewee_migrate INFO : There is nothing to migrate frigate.mqtt INFO : MQTT connected detector.coral_pci INFO : Starting detection process: 50 frigate.edgetpu INFO : Attempting to load TPU as pci frigate.edgetpu INFO : TPU found

This guide is written for using the Corals in Frigate

External Links

https://pve.proxmox.com/wiki/PCI(e)_Passthrough

https://coral.ai/docs/m2/get-started/

https://forum.proxmox.com/threads/pcie-passthrough-atheros-wifi.73204/

https://www.thomas-krenn.com/en/wiki/Enable_Proxmox_PCIe_Passthrough

https://github.com/blakeblackshear/frigate

https://blakeblackshear.github.io/frigate/

If you have any questions just ask.

Regards,

nayneyT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment