Accessing PCI via UIO on T1020/T1040

bradley_gamble · ‎05-15-2019

Hi,

I am currently using a device based on the T10xx reference design (T1022 processor). I have previously been building the Linux kernel using the v1.7 QoriQ SDK (QorIQ-SDK-V1.7-SOURCE-20141218-yocto.iso). however I am looking to upgrade to a recent mainline Linux kernel (In this example, 4.20).

I have successfully built and booted the 4.20 mainline kernel on my device and can access most of the basic devices (ie MTD, GPIO, i2C...) without major issue. However, on the mainline kernel the PCI bus is not being exposed via UIO. Previously on the 3.1x kernel provided with v1.7 QoriQ SDK the following related kernel options were enabled:

# Freescale specific options # CONFIG_FSL_DPA=y CONFIG_FSL_ERRATUM_A_008007=y CONFIG_FSL_IFC=y CONFIG_FSL_MPIC_TIMER_WAKEUP=y CONFIG_FSL_USDPAA=y CONFIG_UIO_FSL_DMA=y

These options appear not to exist in the mainline kernel - I assume they have been renamed or modified since merging back. I have instead enabled what I assume are the equivalent options:

# DMA Devices # CONFIG_FSL_DPA=y CONFIG_FSL_EDMA=y CONFIG_DMA_ENGINE=y
CONFIG_DMA_OF=y
CONFIG_UIO=y
CONFIG_UIO_PCI_GENERIC=y

Booting with these options appears to enumerate the devices. Below is a truncated copy of the V1.7 QoriQ SDK kernel:

fsl-of-dma ffe100300.dma: dma channel dma-uio0-0 initialized
fsl-of-dma ffe100300.dma: dma channel dma-uio0-1 initialized
fsl-of-dma ffe100300.dma: dma channel dma-uio0-2 initialized
fsl-of-dma ffe100300.dma: dma channel dma-uio0-3 initialized
fsl-of-dma ffe100300.dma: dma channel dma-uio0-4 initialized
fsl-of-dma ffe100300.dma: dma channel dma-uio0-5 initialized
fsl-of-dma ffe100300.dma: dma channel dma-uio0-6 initialized
fsl-of-dma ffe100300.dma: dma channel dma-uio0-7 initialized
fsl-of-dma ffe101300.dma: dma channel dma-uio1-0 initialized
fsl-of-dma ffe101300.dma: dma channel dma-uio1-1 initialized
fsl-of-dma ffe101300.dma: dma channel dma-uio1-2 initialized
fsl-of-dma ffe101300.dma: dma channel dma-uio1-3 initialized
fsl-of-dma ffe101300.dma: dma channel dma-uio1-4 initialized
fsl-of-dma ffe101300.dma: dma channel dma-uio1-5 initialized
fsl-of-dma ffe101300.dma: dma channel dma-uio1-6 initialized
fsl-of-dma ffe101300.dma: dma channel dma-uio1-7 initialized

And for comparison, this is the equivalent for the mainline kernel:

Freescale Elo series DMA driver
fsl-elo-dma ffe100300.dma: #0 (fsl,eloplus-dma-channel), irq 28
fsl-elo-dma ffe100300.dma: #1 (fsl,eloplus-dma-channel), irq 29
fsl-elo-dma ffe100300.dma: #2 (fsl,eloplus-dma-channel), irq 30
fsl-elo-dma ffe100300.dma: #3 (fsl,eloplus-dma-channel), irq 31
fsl-elo-dma ffe100300.dma: #4 (fsl,eloplus-dma-channel), irq 76
fsl-elo-dma ffe100300.dma: #5 (fsl,eloplus-dma-channel), irq 77
fsl-elo-dma ffe100300.dma: #6 (fsl,eloplus-dma-channel), irq 78
fsl-elo-dma ffe100300.dma: #7 (fsl,eloplus-dma-channel), irq 79
fsl-elo-dma ffe101300.dma: #0 (fsl,eloplus-dma-channel), irq 32
fsl-elo-dma ffe101300.dma: #1 (fsl,eloplus-dma-channel), irq 33
fsl-elo-dma ffe101300.dma: #2 (fsl,eloplus-dma-channel), irq 34
fsl-elo-dma ffe101300.dma: #3 (fsl,eloplus-dma-channel), irq 35
fsl-elo-dma ffe101300.dma: #4 (fsl,eloplus-dma-channel), irq 80
fsl-elo-dma ffe101300.dma: #5 (fsl,eloplus-dma-channel), irq 81
fsl-elo-dma ffe101300.dma: #6 (fsl,eloplus-dma-channel), irq 82
fsl-elo-dma ffe101300.dma: #7 (fsl,eloplus-dma-channel), irq 83

At this point, I can bind the devices to the uio_pci_generic driver and they appear to be exposed correctly:

~ # ls /dev/uio*
ls: /dev/uio*: No such file or directory
~ # echo "1957 082C" > /sys/bus/pci/drivers/uio_pci_generic/new_id
~ # ls /dev/uio*
/dev/uio0

This appears to work correctly, I can check the values in /sys/class/uio/uio0/ (ie "dev", "version", "uevent") which are identical to that of the V1.7 QoriQ SDK kernel, with the exception of "name" which was "dma-uio0-0" on the V1.7 QoriQ SDK kernel and is "uio-pci-generic" on the mainline kernel.

The main issue for me is that the mainline kernel is missing the /sys/class/uio/uio0/maps/ directory. As a comparison, on the V1.7 QoriQ SDK:

~ # ls -R /sys/class/uio/uio0 /sys/class/uio/uio0: dev event name uevent device maps subsystem version /sys/class/uio/uio0/maps: map0 /sys/class/uio/uio0/maps/map0: addr name offset size

And on the mainline kernel:

~ # ls -R /sys/class/uio/uio0 /sys/class/uio/uio0: dev device event name subsystem uevent version

If I look under /sys/class/dma it appears all the DMa channels are available, but in a different format:

/sys/class/dma # ls -la total 0 drwxr-xr-x 2 root root 0 Jan 1 1970 . drwxr-xr-x 41 root root 0 Jan 1 1970 .. lrwxrwxrwx 1 root root 0 Jan 1 1970 dma0chan0 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan0 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma0chan1 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan1 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma0chan2 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan2 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma0chan3 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan3 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma0chan4 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan4 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma0chan5 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan5 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma0chan6 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan6 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma0chan7 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan7 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma1chan0 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan0 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma1chan1 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan1 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma1chan2 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan2 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma1chan3 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan3 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma1chan4 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan4 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma1chan5 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan5 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma1chan6 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan6 lrwxrwxrwx 1 root root 0 Jan 1 1970 dma1chan7 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan7

We currently use the Freescale source file "dma_driver.c" which exposes the following functions:

int fsl_dma_chan_init(struct dma_ch **dma_ch, uint8_t dma_id, uint8_t ch_id)
int fsl_dma_chan_finish(struct dma_ch *dma_ch)
int fsl_dma_chan_basic_direct_init(struct dma_ch *dma_ch)
int fsl_dma_chan_bwc(struct dma_ch *dma_ch, uint8_t bwc)
int fsl_dma_wait_blocking(struct dma_ch *dma_ch)
int fsl_dma_direct_start(struct dma_ch *dma_ch, dma_addr_t src_phys, dma_addr_t dest_phys, uint32_t len)
int fsl_dma_chain_link_build(struct dma_link_setup_data *link_data, struct dma_link_dsc *link_dsc, uint64_t link_dsc_phys, uint32_t link_count)
int fsl_dma_chain_basic_start(struct dma_ch *dma_ch, struct dma_link_setup_data *link_data, uint64_t link_dsc_phys)

This source file is copyright dated 2011-2012 Freescale Semiconductor, Inc. If there is a newer version of this file, please let me know.

My main questions from this are:

- Am I using the correct driver to access the PCI device? We have software that accesses PCI via UIO so we would like to maintain this compatibility. I assume this related to the "map" directory not being exposed.

- Can I expose the DMA-UIO in the same format as on the v1.7 QoriQ SDK kernel? I assume this was previously done with the Freescale kernel driver - Is it possible to use this on the mainline kernel?

If any more details are required please let me know so that I can provide them.

Thanks in advance!

yipingwang · ‎06-07-2019

Hello Bradley Gamble,

I checked Yocto 2.6 Linux Kernel 4.14 release, DMA UIO driver have been removed from Linux Kernel.

You could add the attached DMA UIO driver file fsl_dma_uio.c in Kernel source folder drivers/uio.

Please add the following in drivers/uio/Makefile.

obj-$(CONFIG_UIO_FSL_DMA) += fsl_dma_uio.o

Please add the following in drivers/uio/Kconfig.

config UIO_FSL_DMA
       tristate "Freescale DMA support"
       depends on FSL_DMA=n
       default n
       help
         UIO driver for the Freescale DMA controller.
         This driver will obtain the DMA nodes from DTB structure, and
         expose the DMA regiter space to user space.

If you don't know what to do here, say N.

Please configure the following in Kernel configuration file, then rebuild Linux Kernel image.

CONFIG_UIO_FSL_DMA=y

Please refer to the following verification log on T2080RDB.

QorIQ SDK (FSL Reference Distro) 2.6 t2080rdb /dev/ttyS0

t2080rdb login: root

root@t2080rdb:~# ls /sys/class/uio/uio0/maps/map0/

addr name offset size

root@t2080rdb:~# ls /sys/class/uio/uio0/

dev device event maps name power subsystem uevent version

root@t2080rdb:~# ls /sys/class/uio/uio0/maps/

map0

root@t2080rdb:~# ls /sys/class/uio/uio0/maps/map0/

addr name offset size

root@t2080rdb:/sys/class/dma# dmesg | grep fsl-of-dma

[ 1.554241] fsl-of-dma ffe100300.dma: dma channel dma-uio0-0 initialized

[ 1.559749] fsl-of-dma ffe100300.dma: dma channel dma-uio0-1 initialized

[ 1.578141] fsl-of-dma ffe100300.dma: dma channel dma-uio0-2 initialized

[ 1.583643] fsl-of-dma ffe100300.dma: dma channel dma-uio0-3 initialized

[ 1.589147] fsl-of-dma ffe100300.dma: dma channel dma-uio0-4 initialized

[ 1.594657] fsl-of-dma ffe100300.dma: dma channel dma-uio0-5 initialized

[ 1.600162] fsl-of-dma ffe100300.dma: dma channel dma-uio0-6 initialized

[ 1.605667] fsl-of-dma ffe100300.dma: dma channel dma-uio0-7 initialized

[ 1.611206] fsl-of-dma ffe101300.dma: dma channel dma-uio1-0 initialized

[ 1.616709] fsl-of-dma ffe101300.dma: dma channel dma-uio1-1 initialized

[ 1.622214] fsl-of-dma ffe101300.dma: dma channel dma-uio1-2 initialized

[ 1.627726] fsl-of-dma ffe101300.dma: dma channel dma-uio1-3 initialized

[ 1.633234] fsl-of-dma ffe101300.dma: dma channel dma-uio1-4 initialized

[ 1.638736] fsl-of-dma ffe101300.dma: dma channel dma-uio1-5 initialized

[ 1.644240] fsl-of-dma ffe101300.dma: dma channel dma-uio1-6 initialized

[ 1.649741] fsl-of-dma ffe101300.dma: dma channel dma-uio1-7 initialized

[ 1.655280] fsl-of-dma ffe102300.dma: dma channel dma-uio2-0 initialized

[ 1.660783] fsl-of-dma ffe102300.dma: dma channel dma-uio2-1 initialized

[ 1.666294] fsl-of-dma ffe102300.dma: dma channel dma-uio2-2 initialized

[ 1.671796] fsl-of-dma ffe102300.dma: dma channel dma-uio2-3 initialized

[ 1.677300] fsl-of-dma ffe102300.dma: dma channel dma-uio2-4 initialized

[ 1.682802] fsl-of-dma ffe102300.dma: dma channel dma-uio2-5 initialized

[ 1.688306] fsl-of-dma ffe102300.dma: dma channel dma-uio2-6 initialized

[ 1.693809] fsl-of-dma ffe102300.dma: dma channel dma-uio2-7 initialized

root@t2080rdb:/sys/class/dma# uname -a

Linux t2080rdb 4.14.67+gf2f35bbfc16b #1 SMP Tue Jun 4 09:40:37 UTC 2019 ppc64 ppc64 ppc64 GNU/Linux

root@t2080rdb:/sys/class/dma#

Thanks,

Yiping

bradley_gamble · ‎06-11-2019

Hi yipingwang‌,

Thank you for your support. I can confirm your suggestion has exposed the "/dev/uio*" paths as required.

The Freescale userspace driver code I am using also requires access to the path "/dev/fsl-usdpaa". This appears to come from the kernel option `FSL_USDPAA` in the Freescale kernel. This driver does not appear to be simple to copy across to the manline kernel.

I have cloned the `usdpaa` repo from Freescale Git, however this code is identical to the userspace code that I am currently using in my application. Is it possible to enable the `fs-usdpaa` driver on the mainline kernel, or have it been superseded by a different interface?

Kind regards,

yipingwang · ‎06-23-2019

USDPAA is not supported in the recent released upstream Kernel.

Kernel configuration options to enable upstream version of DPAA driver for PowerPC as the following.

CONFIG_FSL_DPAA, CONFIG_FSL_FMAN and CONFIG_FSL_DPAA_ETH need to be enabled in kernel menuconfig as below:

Device Drivers --->

SOC (System On Chip) specific Drivers --->

[*] Freescale DPAA 1.x support --->

Device Drivers --->

[*] Network device support --->

[*] Ethernet driver support --->

<*> FMan support

<*> DPAA Ethernet ----

Thanks,

Yiping

Accessing PCI via UIO on T1020/T1040

Accessing PCI via UIO on T1020/T1040

QorIQ T1 Devices