OSDev.org

The Place to Start for Operating System Developers
It is currently Mon Mar 18, 2024 8:13 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: Intel DPDK in virtualization environment
PostPosted: Fri Oct 08, 2021 4:18 am 
Offline
Member
Member

Joined: Sat Dec 22, 2012 12:01 pm
Posts: 73
Hi,

I'm aware of Intel DPDK allows code in usermode running inside a process to interact directly with HW NIC bypassing the (host) kernel/OS. From my understanding DPDK is basically a library designed to be linked against the usermode code.

From an architectural point of view, I believe the model is that the physical NIC is polled by the usermode DPDK PMD (Poll Mode Driver) so there is actually no DMA performed by the NIC itself (i.e. dedicate host Cores poll the NIC and copy the NIC provided packets into userspace buffers provided by the DPDK library/user code).

If the above is correct, I've not however a clear understanding how DPDK model can be employed to exchange packets to/from Virtual Machines (VMs) that runs on the same physical box -- just to fix ideas assume QEM/KVM as virtualization environment running as hypervisor on the same box.

How does the DPDK library running inside its own process copy buffers to/from emulated NICs inside each QEMU hypervisor process ?

Thank you :wink:


Top
 Profile  
 
 Post subject: Re: Intel DPDK in virtualization environment
PostPosted: Fri Oct 08, 2021 11:51 am 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5069
cianfa72 wrote:
From an architectural point of view, I believe the model is that the physical NIC is polled by the usermode DPDK PMD (Poll Mode Driver) so there is actually no DMA performed by the NIC itself (i.e. dedicate host Cores poll the NIC and copy the NIC provided packets into userspace buffers provided by the DPDK library/user code).

Polling means the driver doesn't use IRQs for transmit or receive. The driver can still use DMA.

Routing packets between VMs typically requires the hypervisor to provide a virtual switch and emulate the NICs, but some NICs include a hardware switch.


Top
 Profile  
 
 Post subject: Re: Intel DPDK in virtualization environment
PostPosted: Fri Oct 08, 2021 3:32 pm 
Offline
Member
Member

Joined: Sat Dec 22, 2012 12:01 pm
Posts: 73
Octocontrabass wrote:
Polling means the driver doesn't use IRQs for transmit or receive.
The driver can still use DMA.
ok, so that actually means the driver code may program the DMA-capable NIC to perform DMA to/from buffers provided by PMD driver/DPDK library in userspace.

In that case which is the role of DPDK dedicated CPU Cores since the NIC itself perform the DMA in userspace?


Top
 Profile  
 
 Post subject: Re: Intel DPDK in virtualization environment
PostPosted: Fri Oct 08, 2021 5:21 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5069
DPDK is designed to be as fast as possible. Dedicating a core to each task means those tasks will never have to wait for another task to run, or wait while the OS migrates them to a different core.

This kind of extreme optimization is necessary when handling tens of gigabits of network traffic.


Top
 Profile  
 
 Post subject: Re: Intel DPDK in virtualization environment
PostPosted: Sat Oct 09, 2021 1:04 am 
Offline
Member
Member

Joined: Sat Dec 22, 2012 12:01 pm
Posts: 73
Octocontrabass wrote:
DPDK is designed to be as fast as possible. Dedicating a core to each task means those tasks will never have to wait for another task to run, or wait while the OS migrates them to a different core.
Yes that's definitely makes sense. The point unclear to me is what kind of job the DPDK dedicated Cores actually do.

As said before if the copy of data packets to/from target userspace buffers is done directly by NIC using DMA, then DPDK dedicated Cores's job is just manage and setp the the descriptor'structures, registers ecc.. shared between the PMD driver and DMA-capable NIC ? Thank you.


Top
 Profile  
 
 Post subject: Re: Intel DPDK in virtualization environment
PostPosted: Sat Oct 09, 2021 6:43 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5069
cianfa72 wrote:
then DPDK dedicated Cores's job is just manage and setp the the descriptor'structures, registers ecc.. shared between the PMD driver and DMA-capable NIC ?

The dedicated cores are in charge of receiving packets from one or more Rx queues, performing work on those packets, and transmitting them to one or more Tx queues. The things you've listed are part of the process for receiving and transmitting packets.

For example, a VPN device built around DPDK might use two dedicated cores. One core would receive packets from the local network, decide whether they need encryption, encrypt them as needed, and send them to the internet. Another core would receive packets from the internet, decide which ones need decryption, decrypt them as needed, and send them to the local network.


Top
 Profile  
 
 Post subject: Re: Intel DPDK in virtualization environment
PostPosted: Sun Oct 10, 2021 12:29 am 
Offline
Member
Member

Joined: Sat Dec 22, 2012 12:01 pm
Posts: 73
Octocontrabass wrote:
The dedicated cores are in charge of receiving packets from one or more Rx queues, performing work on those packets, and transmitting them to one or more Tx queues. The things you've listed are part of the process for receiving and transmitting packets.

ok, so basically it is in charge of DPDK dedicated cores to "pullout" packets from Rx queue (Rx ring), process them according some logic, and possibly send/queue them into Tx queue (Tx ring).

Does it make sense ? Thank you.


Top
 Profile  
 
 Post subject: Re: Intel DPDK in virtualization environment
PostPosted: Sun Oct 10, 2021 4:50 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5069
Yes, that's correct.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group