OSDev.org

Hello,
I need to run a networking software from Cisco XRd vRouter within a qemu/kvm VM on my Linux machine.

The requirements specify that Hypervisor/VMM must expose IOMMU support to the guest in order to support SR-IOV/PCI passthrough. However I'm having trouble with this...

XRd vRouter runs as docker container within the guest Linux OS. My understanding is that, from the guest viewpoint, Hypervisor's emulated devices and SR-IOV/PCI passthrough devices are actually indistinguishable. For instance when guest OS performs PCI bus enumeration it retrives the relevant PCI device B/D/F configuration space entries in both cases. It is actually up to the Hypervisor (i.e. qemu) to emulate PCI devices' registers for emulated devices.

Then, for SR-IOV/PCI passthrough devices, the Hypervisor (qemu/kvm) must configure logical processors' hardware IOMMU to remap devices DMA initiated transactions' target GIOVAs to the relevant host machine addresses (HPAs). Note that the guest itself doesn't use IOMMU at all therefore actually target GIOVAs = target GPAs in this case (i.e. Intel scalable-mode PASID is configured to use only one stage of address translation).

Hence the question: why the requirement to expose IOMMU support to the guest to support SR-IOV/PCI passthrough ?

Thanks a lot.

cianfa72 wrote: ↑Mon Jun 09, 2025 6:02 amThe requirements specify that Hypervisor/VMM must expose IOMMU support to the guest in order to support SR-IOV/PCI passthrough.

The host IOMMU is used for PCI passthrough. The guest IOMMU is used for VFIO.

The guest IOMMU is not required, just recommended. You can configure the guest OS to use unsafe no-IOMMU VFIO if you don't want to use the guest IOMMU.

Octocontrabass wrote: ↑Mon Jun 09, 2025 1:30 pm The host IOMMU is used for PCI passthrough.

Ok, yes.

Octocontrabass wrote: ↑Mon Jun 09, 2025 1:30 pmThe guest IOMMU is used for VFIO.
The guest IOMMU is not required, just recommended.

Just to reiterate : "Expose IOMMU to the guest OS" requirement asks the hypervisor to expose IOMMU support to the guest (regardless of whether the physical processors actually support IOMMU or not). Basically the guest is fooled in thinking it is running on a real hardware featuring IOMMU. It is up to the Hypervisor to either fully emulate "guest exposed IOMMU" in software or leverage hardware IOMMU virtualization support (if supported by host physical processor).

Octocontrabass wrote: ↑Mon Jun 09, 2025 1:30 pm You can configure the guest OS to use unsafe no-IOMMU VFIO if you don't want to use the guest IOMMU.

Ok, what are the main tasks of such VFIO driver running in the guest (in both cases, i.e. IOMMU vs no-IOMMU at guest level) ?

Thanks.

cianfa72 wrote: ↑Tue Jun 10, 2025 7:11 amJust to reiterate :

Yes, that's correct.

cianfa72 wrote: ↑Tue Jun 10, 2025 7:11 amOk, what are the main tasks of such VFIO driver running in the guest (in both cases, i.e. IOMMU vs no-IOMMU at guest level) ?

VFIO is basically PCI passthrough but for regular programs instead of VMs. It allows normal userspace programs to directly access hardware devices.

Octocontrabass wrote: ↑Tue Jun 10, 2025 10:20 amVFIO is basically PCI passthrough but for regular programs instead of VMs. It allows normal userspace programs to directly access hardware devices.

Ah ok. So VFIO driver running in the guest OS allows userspace programs/processes (like programs running within the docker container) to access what the guest OS sees as real hardware devices.

Therefore, as far as I can tell, VFIO is really a different feature from VM virtualization (i.e. even on bare-metal systems VFIO driver is needed just to allow userspace programs to directly access hardware devices).

Hence, coming back to the main point, exposing IOMMU to the guest is just recommended to run the safer "IOMMU-based" VFIO drivers within the guest OS.

That all sounds correct to me.