USDPAA installation / activation problem on custom T2081 board

[robwestfall]

We have a custom T2081 based CPU based on the T2080RDB design.  We are getting data packet ordering issues with data coming into the board though the 10G Ethernet ports.  The data shows up, but not in the same order that it was transmitted.  Needless to say our application software is confused. 

DPAA seems to be the way to solve this (hash on source/dest addr/port which would route the data for a particular port to a consistent work queue and the various CPUs will pull in order from that), but I can't figure out how to configure it on the board.   I went through the AN4760 - "Frame Manager Configuration Tool", and now I have two xml files, but no place to put them.

I have read through lots of FTF presentations and App notes and they all talk about the "fmc" app and the "reflector" test program (and other usdpaa test programs), but we don't have them in our custom kernel build and I can't figure out how to build them and get them into the disk image.   A scan of the files in the yocto build directory doesn't show anything.  We have a "/etc/fmc" directory full of files, but no /usr/etc/fmc.

We are running a Yocto 2.3 based kernel (poky-pyro-17.0.1), and have the Freescale meta-freescale layer.  We have USDPAA enabled in the kernel,

This is part of our Kernel config file:

CONFIG_FSL_DPA=y
CONFIG_FSL_DPAA_ADVANCED_DRIVERS=y
CONFIG_FSL_DPAA_CS_THRESHOLD_10G=0x10000000
CONFIG_FSL_DPAA_CS_THRESHOLD_1G=0x06000000
CONFIG_FSL_DPAA_ETH_DEBUGFS=y
CONFIG_FSL_DPAA_ETH_JUMBO_FRAME=y
CONFIG_FSL_DPAA_ETH_MAX_BUF_COUNT=128
CONFIG_FSL_DPAA_ETH_REFILL_THRESHOLD=80
CONFIG_FSL_DPAA_ETH_USE_NDO_SELECT_QUEUE=y
CONFIG_FSL_DPAA_GENERIC_DRIVER=y
CONFIG_FSL_DPAA_HOOKS=y
CONFIG_FSL_DPAA_INGRESS_CS_THRESHOLD=0x10000000
CONFIG_FSL_DPAA_OFFLINE_PORTS=y
CONFIG_FSL_DPA_CAN_WAIT=y
CONFIG_FSL_DPA_CAN_WAIT_SYNC=y
CONFIG_FSL_DPA_PIRQ_FAST=y
CONFIG_FSL_DPA_PIRQ_SLOW=y
CONFIG_FSL_DPA_PORTAL_SHARE=y
CONFIG_FSL_SDK_DPAA_ETH=y
CONFIG_FSL_USDPAA=y
# CONFIG_FSL_DPAA_1588 is not set
# CONFIG_FSL_DPAA_CEETM is not set
# CONFIG_FSL_DPAA_DBG_LOOP is not set
# CONFIG_FSL_DPAA_ETH_DEBUG is not set
# CONFIG_FSL_DPAA_MACSEC is not set
# CONFIG_FSL_DPAA_TS is not set
# CONFIG_FSL_DPA_CHECKING is not set
# CONFIG_FSL_DPA_OFFLOAD is not set
# CONFIG_PTP_1588_CLOCK_DPAA is not set

We have also added the following packages into our custom layer
#### Packages for DPAA
IMAGE_INSTALL_append = " dpa-offload"
IMAGE_INSTALL_append = " eth-config"
IMAGE_INSTALL_append = " flib"
IMAGE_INSTALL_append = " fmlib"
IMAGE_INSTALL_append = " fm-ucode"
IMAGE_INSTALL_append = " usdpaa"

What am I missing?

Thanks,

Rob Westfall

[yipingwang]

Hello ROB WESTFALL,

Please add the following packages in rootfs through IMAGE_INSTALL_append.

    fmc \
    usdpaa \
    usdpaa-apps \

USDPAA related xml configuration files will be integrated in /usr/etc/.

For data packet disordering under Linux, you could use core-affined queue to implement order order preservation, please use one of the following default FMC policy files.
/etc/fmc/config/private/t2080rdb/*/policy_ipv4.xml
/etc/fmc/config/private/t2080rdb/*/policy_ipv6.xml

Please execute FMC commands as the following before all networking related operation.
# fmc -c /etc/fmc/config/private/t2080rdb/*/config.xml -p /etc/fmc/config/private/t2080rdb/*/policy_ipv4.xml -a

This is because in DPAA1, each interface used by default one pool channel
across all software portals and also the dedicated channel of each CPU. In
Linux Kernel, PCD Frame queues in use dedicated channels.You could refer to
the section "5. Dedicated and Pool Channels Usage in Linux Kernel" in
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity
.nxp.com%2Fdocs%2FDOC-329916&data=02%7C01%7Ctech.support%40nxp.com%7Cad3ec3502cc849493def08d5f
76ab1a1%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636686959995990555&
sdata=rcGwOiK4On8o9UmHqAqgnCiADxtT%2BkDTpiAFnwSiV88%3D&reserved=0 for
details.

In addition, you need to use multiple flows, after executing FMC policy, one flow will bind to one core, so all 8 cores will be used by multiple flows. In the real scenario, one user application uses one flow.

Have a great day,
TIC

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[robwestfall]

Things are a little bit better. 

I finally have the fmc and ipfwd_app executables in my system and I can configure things.

But it still doesn't work - I can't ping from my board to the 10G network switch.  I got it to work once, but I can't get it to work again.

I configure the system with the Frame Manager Config Tool:
target:~# /usr/bin/fmc -c /etc/fmc/config/config.xml -p /etc/fmc/config/policy_ipv4.xml -a

Then I run a DPAA demo APP:
target:~# /usr/bin/ipfwd_app 7 -b 1600:1600:1600  -c /etc/fmc/config/config.xml -p /etc/fmc/config/policy_ipv4.xml -d 0x10000000  --non-interactive &

Found /fsl,dpaa/ethernet@8, Tx Channel = 800, FMAN = 0, Port ID = 9
QMan: Allocated lookup table at 0x3fff83e39010, entry count 32769
Configuring for 1 network interface
Allocated DMA region size 0x10000000
cpu hotplug daemon not running: No such file or directory
ipfwd_app starting
Message queue to send: /mq_snd_15684
Message queue to receive: /mq_rcv_15684
Configured 10G port @ FMAN:0, MAC:9 as IF_IDX:89
Released 1600 bufs to BPID 17
Data stashing: Enabled
Thread uid:0 alive (on cpu 7)

I see buffers in the bman pool:
target:~# cat /sys/devices/platform/ffe000000.soc/ffe31a000.bman/pool_count/32
1600
target:~# cat /sys/devices/platform/ffe000000.soc/ffe31a000.bman/pool_count/32
863

The system is configured for a share ethernet driver on the 10G port.  The two 1G ports are not DPAA enabled:

target:~# ls -al /sys/devices/platform/fsl,dpaa/fsl\,dpaa\:ethernet\@*/driver
lrwxrwxrwx    1 root     root             0 Oct  4 17:52 /sys/devices/platform/fsl,dpaa/fsl,dpaa:ethernet@2/driver -> ../../../../bus/platform/drivers/fsl_dpa
lrwxrwxrwx    1 root     root             0 Oct  4 17:52 /sys/devices/platform/fsl,dpaa/fsl,dpaa:ethernet@3/driver -> ../../../../bus/platform/drivers/fsl_dpa
lrwxrwxrwx    1 root     root             0 Oct  4 17:52 /sys/devices/platform/fsl,dpaa/fsl,dpaa:ethernet@8/driver -> ../../../../bus/platform/drivers/fsl_dpa_shared-shared

The Kernel see the DPAA configuration and sets stuff up:
target:~# journalctl |egrep -i "(fsl|dpaa|fmc)"
Oct 04 17:52:28 t2080rdb-64b kernel: Reserved memory: initialized node bman-fbpr, compatible id fsl,bman-fbpr
Oct 04 17:52:28 t2080rdb-64b kernel: Reserved memory: initialized node qman-fqd, compatible id fsl,qman-fqd
Oct 04 17:52:28 t2080rdb-64b kernel: Reserved memory: initialized node qman-pfdr, compatible id fsl,qman-pfdr
Oct 04 17:52:28 t2080rdb-64b kernel: Reserved memory: initialized node usdpaa_mem, compatible id fsl,usdpaa-mem
Oct 04 17:52:28 t2080rdb-64b kernel: Reserved memory: initialized node pme-pdsr, compatible id fsl,pme-pdsr
Oct 04 17:52:28 t2080rdb-64b kernel: Reserved memory: initialized node pme-sre, compatible id fsl,pme-sre
Oct 04 17:52:28 t2080rdb-64b kernel: USDPAA region at e0000000:10000000(e0000:10000), 1 TLB1 entries)
Oct 04 17:52:28 t2080rdb-64b kernel: fsl-ifc ffe124000.localbus: Freescale Integrated Flash Controller
Oct 04 17:52:28 t2080rdb-64b kernel: fsl-ifc ffe124000.localbus: IFC version 1.3, 8 banks
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_mac: mac.c:458:mac_load() fsl_mac: FSL FMan MAC API based driver
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_mac ffe4e4000.ethernet: FMan MEMAC
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_mac ffe4e4000.ethernet: FMan MAC address: 12:34:12:38:9a:b0
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_mac ffe4e6000.ethernet: FMan MEMAC
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_mac ffe4e6000.ethernet: FMan MAC address: 12:34:12:38:9a:b0
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_mac ffe4f0000.ethernet: FMan MEMAC
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_mac ffe4f0000.ethernet: FMan MAC address: 12:34:12:38:9a:b0
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_mac ffe4f2000.ethernet: FMan MEMAC
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_mac ffe4f2000.ethernet: FMan MAC address: 12:34:12:38:9a:b0
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_dpa: dpaa_eth.c:1163:dpa_load() FSL DPAA Ethernet driver
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_dpa: FSL DPAA Ethernet debugfs entries
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_dpa: dpaa_eth.c:1089:dpaa_eth_priv_probe() fsl_dpa: Probed interface eth0
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_dpa: dpaa_eth.c:1089:dpaa_eth_priv_probe() fsl_dpa: Probed interface eth1
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_advanced: dpaa_eth_base.c:251:dpa_advanced_load() FSL DPAA Advanced drivers:
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_proxy: dpaa_eth_proxy.c:354:dpa_proxy_load() FSL DPAA Proxy initialization driver
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_dpa_shared: dpaa_eth_shared.c:885:dpa_shared_load() FSL DPAA Shared Ethernet driver
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_dpa_shared: dpaa_eth_shared.c:833:dpaa_eth_shared_probe() fsl_dpa_shared: Probed shared interface eth2
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_dpa_macless: dpaa_eth_macless.c:469:dpa_macless_load() FSL DPAA MACless Ethernet driver
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_oh: offline_port.c:823:oh_port_load() FSL FMan Offline Parsing port driver
Oct 04 17:52:28 t2080rdb-64b kernel: Freescale USDPAA process driver
Oct 04 17:52:28 t2080rdb-64b kernel: Freescale USDPAA process IRQ driver
Oct 04 17:52:28 t2080rdb-64b kernel: fsl-dce ffe312000.dce: Device /soc@ffe000000/dce@312000 initialized ver: 0x0af00101
Oct 04 17:52:28 t2080rdb-64b kernel: fsl-pme ffe316000.pme: ver: 0x00100201
Oct 04 17:52:28 t2080rdb-64b kernel: fsl-pme2-scan: device pme_scan registered
Oct 04 17:52:28 t2080rdb-64b kernel: fsl-hv: no hypervisor found
Oct 04 17:52:28 t2080rdb-64b kernel: fsl_generic: FSL DPAA Generic Ethernet driver

What am I missing?

Rob Westfall