1). 簡(jiǎn)介

DPDK (Data Plane Development Kit) 軟件是一組用戶空間庫(kù)和驅(qū)動(dòng)程序，可加速在所有主要 CPU 架構(gòu)上運(yùn)行的網(wǎng)絡(luò)數(shù)據(jù)包處理工作負(fù)載，以便提升整個(gè)網(wǎng)絡(luò)數(shù)據(jù)服務(wù)的QoS。其最早由 Intel 大約 2010年創(chuàng)建，后由6WIND公司發(fā)展為開(kāi)源社區(qū)版本后，極大的拓展了DPDK的影響力，現(xiàn)在DPDI已經(jīng)是是 Linux 基金會(huì)旗下的開(kāi)源項(xiàng)目之一，其支持涵蓋了主流的CPU架構(gòu)和繁多的NIC驅(qū)動(dòng)以及虛擬化平臺(tái)支持。

DPDK使用了輪詢 (polling) 而不是中斷來(lái)處理數(shù)據(jù)包。在收到數(shù)據(jù)包時(shí)，經(jīng) DPDK 重載的網(wǎng)卡驅(qū)動(dòng)不會(huì)通過(guò)中斷通知 CPU ，而是直接將數(shù)據(jù)包存入內(nèi)存，交付應(yīng)用層軟件通過(guò) DPDK 提供的接口來(lái)直接處理，這樣節(jié)省了大量的 CPU 中斷時(shí)間和內(nèi)存拷貝時(shí)間。

本文所要測(cè)試的 NXP iMX8M Mini ARM SoC平臺(tái)集成的 FEC NIC 在 DPDK 21.11 LTS發(fā)布版本提供了相關(guān)支持， 具體的 DPDK ENETFEC Poll Mode 驅(qū)動(dòng)描述以及功能支持情況請(qǐng)見(jiàn)如下 dpdk.org 文檔，本文就對(duì)其進(jìn)行簡(jiǎn)單部署測(cè)試。

http://doc.dpdk.org/guides/nics/enetfec.html

本文所演示的平臺(tái)來(lái)自于Toradex Verdin iMX8M Mini 嵌入式平臺(tái)，基于 NXP iMX8M Mini 系列 ARM 處理器，核心為 Cortex-A53，包含一個(gè)支持 DPDK PMD 驅(qū)動(dòng)的以太網(wǎng)接口。

2). 硬件準(zhǔn)備

a). DUT-A : Verdin iMX8MM ARM 核心版配合 Dahlia Carrier Board 載板，并連接調(diào)試串口，后續(xù)測(cè)試中作為DPDK Packet Generator 設(shè)備。

b). DUT-B : Verdin iMX8MM ARM 核心版配合 Verdin Development Board 載板，并連接調(diào)試串口，后續(xù)測(cè)試中作為DPDK Packet Generator 設(shè)備。

c). Dahlia 載板 X11千兆網(wǎng)口和 Verdin Development Board X25 千兆網(wǎng)口通過(guò)一根網(wǎng)線直通連接。

3). 軟件準(zhǔn)備

a). Toradex Verdin iMXMM 模塊標(biāo)準(zhǔn) Ycoto Linux BSP 中沒(méi)有包含 DPDK 相關(guān)庫(kù)和驅(qū)動(dòng)，需要通過(guò) Ycoto Project/Openembedded 編譯環(huán)境來(lái)進(jìn)行添加編譯，生成新的測(cè)試 Linux BSP Image。

b). 參考 Ycoto 編譯環(huán)境來(lái)下載配置相關(guān)編譯環(huán)境，使用最新的針對(duì) Ycoto Linux BSP 6/Linux Kernel 5.15.x 版本的 branch

---------------------------------------

$ mkdir ${HOME}/oe-core

$ cd ${HOME}/oe-core

$ repo init -u git://git.toradex.com/toradex-manifest.git -b kirkstone-6.x.y -m tdxref/default.xml

$ repo sync

---------------------------------------

c). 由于默認(rèn)的 oe-core/layers/meta-freescale/recipes-extended/dpdk/ 提供的 DPDK 19.11版本還未支持 iMX8MM，需要參考 NXP i.MX_Yocto_Project_User's_Guide 文檔，下載 NXP imx-6.1.1-1.0.0 branch 代碼來(lái)獲取 DPDK 21.11 版本的 BB 描述文件。

---------------------------------------

$ mkdir ${HOME}/imx-yocto-bsp

$ cd ${HOME}/imx-yocto-bsp

$ repo init -u https://github.com/nxp-imx/imx-manifest -b imx-linux-langdale -m imx-6.1.1-1.0.0.xml

$ repo sync

---------------------------------------

d). 將 NXP imx-6.1.1-1.0.0 branch 包含的 DPDK 21.11 BB 文件復(fù)制到 Toradex Ycoto 環(huán)境

---------------------------------------

$ cd ${HOME}/oe-core

$ mkdir -p layers/meta-toradex-nxp/recipes-extended/dpdk

$ cp ${HOME}/imx-ycoto-bsp_6.1.1-1.0.0/sources/meta-imx/meta-sdk/recipes-extended/dpdk/dpdk* layers/meta-toradex-nxp/recipes-extended/dpdk/

$ $ ls layers/meta-toradex-nxp/recipes-extended/dpdk/

dpdk_21.11.bb  dpdk.inc

---------------------------------------

e). 修改配置文件后編譯測(cè)試 Ycoto Linux BSP Image 文件

---------------------------------------

### Add below to ${HOME}/oe-core/build/conf/local.conf ###

# add dpdk package

IMAGE_INSTALL:append = " dpdk dpdk-examples dpdk-tools"

# accept freescale license

ACCEPT_FSL_EULA = "1"

### Compile BSP Image ###

$ MACHINE=verdin-imx8mm bitbake tdx-reference-multimedia-image

---------------------------------------

f). 參考這里的說(shuō)明將編譯生成的 Ycoto Linux Image 通過(guò) Toradex Easy Installer 更新到兩個(gè)測(cè)試平臺(tái)的 Verdin iMX8MM 模塊

g). 由于默認(rèn) Linux BSP 網(wǎng)口驅(qū)動(dòng)還是 NXP 標(biāo)準(zhǔn) Linux FEC Kernel驅(qū)動(dòng)，需要通過(guò)修改 Linux Kernel Device Tree 來(lái)適配為 DPDK PMD 驅(qū)動(dòng)。

./ 參考這里的說(shuō)明下載 Linux Kernel 源代碼， 對(duì)應(yīng) Branch 為 toradex_5.15-2.1.x-imx

./ 參考如下 Patch 文件修改 Device Tree 源代碼，并重新編譯生成所需的 Device Tree Binary 文件 imx8mm-verdin-wifi-dev.dtb

---------------------------------------

diff --git a/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts b/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts

index ef952021832e..b2d60695ecf9 100644

--- a/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts

+++ b/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts

@@ -16,3 +16,7 @@

                     "toradex,verdin-imx8mm",

                     "fsl,imx8mm";

 };

+

+&fec1 {

+        compatible = "fsl,imx8mm-fec-uio";

+};

---------------------------------------

./ 將新的 imx8mm-verdin-wifi-dev.dtb 替換兩個(gè)平臺(tái)的 Verdin iMX8MM Linux /boot 下默認(rèn)使用的 Device Tree 文件并重新啟動(dòng)，查看原本的 eth0 接口已經(jīng)沒(méi)有 probe 了

---------------------------------------

root@verdin-imx8mm-07276322:~# dmesg |grep eth

[    0.000000] psci: probing for conduit method from DT.

[    1.716513] fec-uio 30be0000.ethernet: UIO device full name imx-fec-uio initialized

[    8.382278] using random self ethernet address

[    8.387105] using random host ethernet address

root@verdin-imx8mm-07276322:~# ifconfig -a

can0: flags=128 mtu 16

        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 10  (UNSPEC)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        device interrupt 78  

lo: flags=73 mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 188  bytes 13742 (13.4 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 188  bytes 13742 (13.4 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

...

---------------------------------------

4). DPDK 示例應(yīng)用測(cè)試

a). 首先配置 Huge Pages，可以通過(guò)設(shè)置下面 U-boot 環(huán)境變量來(lái)配置，這樣就無(wú)需每次啟動(dòng)都配置一次

---------------------------------------

### configure huge paces ###

$ fw_setenv tdxargs ‘default_hugepagesz=2m hugepagesz=2m hugepages=448’

$ reboot

### check huge pages status ###

root@verdin-imx8mm-07276322:~# cat /proc/meminfo |grep -i '^hugepages'                          

HugePages_Total:     448

HugePages_Free:      448

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

---------------------------------------

b). 示例測(cè)試1 ： dpdk-testpmd 示例應(yīng)用測(cè)試

./ dpdk-testpmd 應(yīng)用配合的參數(shù)說(shuō)明可以參考這里說(shuō)明

./ DUT-B 通過(guò) dpdk-testpmd 示例應(yīng)用 Rx 模式作為 Packet Receiver

---------------------------------------

### DUT-B configure to Rx Mode ###

root@verdin-imx8mm-07275765:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=rxonly                                                                   

EAL: Detected CPU lcores: 4

EAL: Detected NUMA nodes: 1

EAL: Detected static linkage of DPDK

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: lsx_pciep:SoC(0xcc000000) not supported

EAL: Scan for (lsx_pciep) bus failed.

EAL: Selected IOVA mode 'PA'

EAL: No available 32768 kB hugepages reported

EAL: No available 64 kB hugepages reported

EAL: No available 1048576 kB hugepages reported

EAL: VFIO support initialized

TELEMETRY: No legacy callbacks, legacy socket not created

Interactive-mode selected

Set rxonly packet forwarding mode

Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config p

arameters along with --numa.

testpmd: create a new mbuf pool: n=171456, size=2176, socket=0

testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)

Port 0: 01:01:01:01:01:01

Checking link statuses...

Done

### start forward mode ###

testpmd> start

rxonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation m

ode: native

Logical Core 1 (socket 0) forwards packets on 1 streams:

  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  rxonly packet forwarding packets/burst=32

  nb forwarding cores=1 - nb forwarding ports=1

  port 0: RX queue number: 1 Tx queue number: 1

    Rx offloads=0x0 Tx offloads=0x0

    RX queue: 0

      RX desc=0 - RX free threshold=0

      RX threshold registers: pthresh=0 hthresh=0  wthresh=0

      RX Offloads=0x0

    TX queue: 0

      TX desc=0 - TX free threshold=0

      TX threshold registers: pthresh=0 hthresh=0  wthresh=0

      TX offloads=0x0 - TX RS bit threshold=0

---------------------------------------

./ DUT-A 通過(guò) dpdk-testpmd 示例應(yīng)用 Tx 模式作為 Packet Generator

---------------------------------------

### DUT-A configure to Tx Mode ###

root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=txonly

EAL: Detected CPU lcores: 4

EAL: Detected NUMA nodes: 1

EAL: Detected static linkage of DPDK

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: lsx_pciep:SoC(0xcc000000) not supported

EAL: Scan for (lsx_pciep) bus failed.

EAL: Selected IOVA mode 'PA'

EAL: No available 32768 kB hugepages reported

EAL: No available 64 kB hugepages reported

EAL: No available 1048576 kB hugepages reported

EAL: VFIO support initialized

TELEMETRY: No legacy callbacks, legacy socket not created

Interactive-mode selected

Set txonly packet forwarding mode

Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config p

arameters along with --numa.

testpmd: create a new mbuf pool: n=171456, size=2176, socket=0

testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)

Port 0: 01:01:01:01:01:01

Checking link statuses...

Done

### start forward mode ###

testpmd> start

txonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation m

ode: native

Logical Core 1 (socket 0) forwards packets on 1 streams:

  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  txonly packet forwarding packets/burst=32

  packet len=64 - nb packet segments=1

  nb forwarding cores=1 - nb forwarding ports=1

  port 0: RX queue number: 1 Tx queue number: 1

    Rx offloads=0x0 Tx offloads=0x0

    RX queue: 0

      RX desc=0 - RX free threshold=0

      RX threshold registers: pthresh=0 hthresh=0  wthresh=0

      RX Offloads=0x0

    TX queue: 0

      TX desc=0 - TX free threshold=0

      TX threshold registers: pthresh=0 hthresh=0  wthresh=0

      TX offloads=0x0 - TX RS bit threshold=0

---------------------------------------

./ 測(cè)試一段時(shí)間后按順序分別停止 DUT-A 和 DUT-B，查看統(tǒng)計(jì)結(jié)果，顯示收發(fā)一致，工作正常

---------------------------------------

### Tx statistic on DUT-A ###

testpmd> stop

Telling cores to stop...

Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 23614688       TX-dropped: 0             TX-total: 23614688

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 23614688       TX-dropped: 0             TX-total: 23614688

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

### Rx statistic on DUT-B ###

testpmd> stop

Telling cores to stop...

Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 23614688       RX-dropped: 0             RX-total: 23614688

  TX-packets: 0              TX-dropped: 0             TX-total: 0

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 23614688       RX-dropped: 0             RX-total: 23614688

  TX-packets: 0              TX-dropped: 0             TX-total: 0

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

---------------------------------------

c). 示例測(cè)試2 ： dpdk-testpmd Tx Mode 配合 dpdk-l2fwd 示例應(yīng)用測(cè)試

./ dpdk-l2fwd 應(yīng)用配合的參數(shù)說(shuō)明可以參考這里說(shuō)明

./ DUT-B 通過(guò) dpdk-l2fwd 配置為 L2 forward 模式

---------------------------------------

root@verdin-imx8mm-07275765:~# /usr/share/dpdk/examples/dpdk-l2fwd -c 0x1 -n 1 --vdev 'net_enetfec' -- -p 0x1 -P

---------------------------------------

// 注意使用 -P 參數(shù)來(lái)接受全部 MAC Address的包，否則不屬于本機(jī)的會(huì)被自動(dòng)丟棄

./ DUT-A 如示例 1 通過(guò) dpdk-testpmd 示例應(yīng)用 Tx 模式作為 Packet Generator 并 start 開(kāi)始發(fā)包測(cè)試

---------------------------------------

root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --txd=1 --burst=1 --forward-mode=txonly

...

testpmd> start

...

---------------------------------------

// 注意，使用 --txd 和 --burst 來(lái)控制發(fā)包的速率，太快則 dpdk-l2fwd 無(wú)法及時(shí)處理

./ 測(cè)試一段時(shí)間后停止 DUT-A 發(fā)包，查看統(tǒng)計(jì)結(jié)果，顯示收發(fā)一致，同時(shí) DUT-B L2 IO 轉(zhuǎn)發(fā)也一致，工作正常

---------------------------------------

### Tx statistic on DUT-A ###

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 13839386       TX-dropped: 41509766      TX-total: 55349152

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 13839386       TX-dropped: 83019532      TX-total: 96858918

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

### IO statistic on DUT-B ###

Port statistics ====================================

Statistics for port 0 ------------------------------

Packets sent:                 13839386

Packets received:             13839386

Packets dropped:                     0

Aggregate statistics ===============================

Total packets sent:           13839386

Total packets received:       13839386

Total packets dropped:               0

====================================================

---------------------------------------

d). 示例測(cè)試3 ： dpdk-testpmd IO Mode 示例應(yīng)用測(cè)試

./ DUT-B 通過(guò) dpdk-testpmd 配置為 IO forward 模式并 start 開(kāi)始準(zhǔn)備轉(zhuǎn)發(fā)從 DUT-A 發(fā)送的包

---------------------------------------

root@verdin-imx8mm-07275765:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=io

...

testpmd> start

...

---------------------------------------

./ DUT-A 通過(guò) dpdk-testpmd 配置為 IO forward 模式并通過(guò) start tx_first開(kāi)始進(jìn)行先發(fā)一個(gè)包然后再 IO 收包的測(cè)試

---------------------------------------

root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=io

...

testpmd> start tx_first

...

---------------------------------------

./ 測(cè)試一段時(shí)間后按順序分別停止 DUT-A 和 DUT-B，查看統(tǒng)計(jì)結(jié)果，DUT-A 因?yàn)橄劝l(fā)一組 Burst 包（ Burst 數(shù)值默認(rèn)是32），因此 Tx 比 Rx 多32；同時(shí) DUT-B IO 轉(zhuǎn)發(fā)一致且和 DUT-A Tx 一致，工作正常

---------------------------------------

### IO statistic on DUT-A ###

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 10381092       RX-dropped: 0             RX-total: 10381092

  TX-packets: 10381124       TX-dropped: 0             TX-total: 10381124

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 10381092       RX-dropped: 0             RX-total: 10381092

  TX-packets: 10381124       TX-dropped: 0             TX-total: 10381124

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

### IO statistic on DUT-B ###

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 10381124       RX-dropped: 0             RX-total: 10381124

  TX-packets: 10381124       TX-dropped: 0             TX-total: 10381124

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 10381124       RX-dropped: 0             RX-total: 10381124

  TX-packets: 10381124       TX-dropped: 0             TX-total: 10381124

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

---------------------------------------

5). 總結(jié)

本文基于NXP iMX8M Mini系列處理器簡(jiǎn)單演示了 DPDK 軟件和驅(qū)動(dòng)的部署以及示例應(yīng)用測(cè)試，按照 NXP 官方說(shuō)明， iMX8M Plus平臺(tái) ETH1 應(yīng)該也支持 DPDK，只是目前在 toradex_5.15-2.0.x-imx branch 下驅(qū)動(dòng)支持還存在一定問(wèn)題，后續(xù)更高版本 branch 可能能夠解決。

參考文檔

NXP i.MX_Linux_Reference_Manual  Chapter 10 Data Plane Development Kit (DPDK)

http://doc.dpdk.org/guides/nics/enetfec.html