Accessing PCI via UIO on T1020/T1040

[bradley_gamble]

Hi,

I am currently using a device based on the T10xx reference design (T1022 processor). I have previously been building the Linux kernel using the v1.7 QoriQ SDK (QorIQ-SDK-V1.7-SOURCE-20141218-yocto.iso). however I am looking to upgrade to a recent mainline Linux kernel (In this example, 4.20).

I have successfully built and booted the 4.20 mainline kernel on my device and can access most of the basic devices (ie MTD, GPIO, i2C...) without major issue. However, on the mainline kernel the PCI bus is not being exposed via UIO. Previously on the 3.1x kernel provided with v1.7 QoriQ SDK the following related kernel options were enabled:

# Freescale specific options # CONFIG_FSL_DPA=y CONFIG_FSL_ERRATUM_A_008007=y CONFIG_FSL_IFC=y CONFIG_FSL_MPIC_TIMER_WAKEUP=y CONFIG_FSL_USDPAA=y CONFIG_UIO_FSL_DMA=y

These options appear not to exist in the mainline kernel - I assume they have been renamed or modified since merging back. I have instead enabled what I assume are the equivalent options:

# DMA Devices # CONFIG_FSL_DPA=y CONFIG_FSL_EDMA=y CONFIG_DMA_ENGINE=y
CONFIG_DMA_OF=y
CONFIG_UIO=y
CONFIG_UIO_PCI_GENERIC=y

Booting with these options appears to enumerate the devices. Below is a truncated copy of the V1.7 QoriQ SDK kernel:

fsl-of-dma ffe100300.dma: dma channel dma-uio0-0 initialized

fsl-of-dma ffe100300.dma: dma channel dma-uio0-1 initialized

fsl-of-dma ffe100300.dma: dma channel dma-uio0-2 initialized

fsl-of-dma ffe100300.dma: dma channel dma-uio0-3 initialized

fsl-of-dma ffe100300.dma: dma channel dma-uio0-4 initialized

fsl-of-dma ffe100300.dma: dma channel dma-uio0-5 initialized

fsl-of-dma ffe100300.dma: dma channel dma-uio0-6 initialized

fsl-of-dma ffe100300.dma: dma channel dma-uio0-7 initialized

fsl-of-dma ffe101300.dma: dma channel dma-uio1-0 initialized

fsl-of-dma ffe101300.dma: dma channel dma-uio1-1 initialized

fsl-of-dma ffe101300.dma: dma channel dma-uio1-2 initialized

fsl-of-dma ffe101300.dma: dma channel dma-uio1-3 initialized

fsl-of-dma ffe101300.dma: dma channel dma-uio1-4 initialized

fsl-of-dma ffe101300.dma: dma channel dma-uio1-5 initialized

fsl-of-dma ffe101300.dma: dma channel dma-uio1-6 initialized

fsl-of-dma ffe101300.dma: dma channel dma-uio1-7 initialized

And for comparison, this is the equivalent for the mainline kernel:

Freescale Elo series DMA driver

fsl-elo-dma ffe100300.dma: #0 (fsl,eloplus-dma-channel), irq 28

fsl-elo-dma ffe100300.dma: #1 (fsl,eloplus-dma-channel), irq 29

fsl-elo-dma ffe100300.dma: #2 (fsl,eloplus-dma-channel), irq 30

fsl-elo-dma ffe100300.dma: #3 (fsl,eloplus-dma-channel), irq 31

fsl-elo-dma ffe100300.dma: #4 (fsl,eloplus-dma-channel), irq 76

fsl-elo-dma ffe100300.dma: #5 (fsl,eloplus-dma-channel), irq 77

fsl-elo-dma ffe100300.dma: #6 (fsl,eloplus-dma-channel), irq 78

fsl-elo-dma ffe100300.dma: #7 (fsl,eloplus-dma-channel), irq 79

fsl-elo-dma ffe101300.dma: #0 (fsl,eloplus-dma-channel), irq 32

fsl-elo-dma ffe101300.dma: #1 (fsl,eloplus-dma-channel), irq 33

fsl-elo-dma ffe101300.dma: #2 (fsl,eloplus-dma-channel), irq 34

fsl-elo-dma ffe101300.dma: #3 (fsl,eloplus-dma-channel), irq 35

fsl-elo-dma ffe101300.dma: #4 (fsl,eloplus-dma-channel), irq 80

fsl-elo-dma ffe101300.dma: #5 (fsl,eloplus-dma-channel), irq 81

fsl-elo-dma ffe101300.dma: #6 (fsl,eloplus-dma-channel), irq 82

fsl-elo-dma ffe101300.dma: #7 (fsl,eloplus-dma-channel), irq 83

At this point, I can bind the devices to the uio_pci_generic driver and they appear to be exposed correctly:

~ # ls /dev/uio*

ls: /dev/uio*: No such file or directory

~ # echo "1957 082C" > /sys/bus/pci/drivers/uio_pci_generic/new_id

~ # ls /dev/uio*

/dev/uio0

This appears to work correctly, I can check the values in /sys/class/uio/uio0/ (ie "dev", "version", "uevent") which are identical to that of the V1.7 QoriQ SDK kernel, with the exception of "name" which was "dma-uio0-0" on the V1.7 QoriQ SDK kernel and is "uio-pci-generic" on the mainline kernel.

The main issue for me is that the mainline kernel is missing the /sys/class/uio/uio0/maps/ directory. As a comparison, on the V1.7 QoriQ SDK:

~ # ls -R /sys/class/uio/uio0
/sys/class/uio/uio0:
dev        event      name       uevent
device     maps       subsystem  version

/sys/class/uio/uio0/maps:
map0

/sys/class/uio/uio0/maps/map0:
addr    name    offset  size

And on the mainline kernel:

~ # ls -R /sys/class/uio/uio0
/sys/class/uio/uio0:
dev        device     event      name       subsystem  uevent     version

If I look under /sys/class/dma it appears all the DMa channels are available, but in a different format:

/sys/class/dma # ls -la
total 0
drwxr-xr-x    2 root     root             0 Jan  1  1970 .
drwxr-xr-x   41 root     root             0 Jan  1  1970 ..
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma0chan0 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan0
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma0chan1 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan1
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma0chan2 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan2
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma0chan3 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan3
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma0chan4 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan4
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma0chan5 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan5
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma0chan6 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan6
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma0chan7 -> ../../devices/platform/ffe000000.soc/ffe100300.dma/dma/dma0chan7
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma1chan0 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan0
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma1chan1 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan1
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma1chan2 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan2
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma1chan3 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan3
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma1chan4 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan4
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma1chan5 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan5
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma1chan6 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan6
lrwxrwxrwx    1 root     root             0 Jan  1  1970 dma1chan7 -> ../../devices/platform/ffe000000.soc/ffe101300.dma/dma/dma1chan7

We currently use the Freescale source file "dma_driver.c" which exposes the following functions:

• int fsl_dma_chan_init(struct dma_ch **dma_ch, uint8_t dma_id, uint8_t ch_id)
• int fsl_dma_chan_finish(struct dma_ch *dma_ch)
• int fsl_dma_chan_basic_direct_init(struct dma_ch *dma_ch)
• int fsl_dma_chan_bwc(struct dma_ch *dma_ch, uint8_t bwc)
• int fsl_dma_wait_blocking(struct dma_ch *dma_ch)
• int fsl_dma_direct_start(struct dma_ch *dma_ch, dma_addr_t src_phys, dma_addr_t dest_phys, uint32_t len)
• int fsl_dma_chain_link_build(struct dma_link_setup_data *link_data, struct dma_link_dsc *link_dsc, uint64_t link_dsc_phys, uint32_t link_count)
• int fsl_dma_chain_basic_start(struct dma_ch *dma_ch, struct dma_link_setup_data *link_data, uint64_t link_dsc_phys)

This source file is copyright dated 2011-2012 Freescale Semiconductor, Inc. If there is a newer version of this file, please let me know.

My main questions from this are:

- Am I using the correct driver to access the PCI device? We have software that accesses PCI via UIO so we would like to maintain this compatibility. I assume this related to the "map" directory not being exposed.

- Can I expose the DMA-UIO in the same format as on the v1.7 QoriQ SDK kernel? I assume this was previously done with the Freescale kernel driver - Is it possible to use this on the mainline kernel?

If any more details are required please let me know so that I can provide them.

Thanks in advance!

[yipingwang]

Hello Bradley Gamble,

I checked Yocto 2.6 Linux Kernel 4.14 release, DMA UIO driver have been removed from Linux Kernel.

You could add the attached DMA UIO driver file fsl_dma_uio.c in Kernel source folder drivers/uio.

Please add the following in drivers/uio/Makefile.

obj-$(CONFIG_UIO_FSL_DMA)      += fsl_dma_uio.o

Please add the following in drivers/uio/Kconfig.

config UIO_FSL_DMA
       tristate "Freescale DMA support"
       depends on FSL_DMA=n
       default n
       help
         UIO driver for the Freescale DMA controller.
         This driver will obtain the DMA nodes from DTB structure, and
         expose the DMA regiter space to user space.

         If you don't know what to do here, say N.

Please configure the following in Kernel configuration file, then rebuild Linux Kernel image.

CONFIG_UIO_FSL_DMA=y

Please refer to the following verification log on T2080RDB.

QorIQ SDK (FSL Reference Distro) 2.6 t2080rdb /dev/ttyS0

t2080rdb login: root

root@t2080rdb:~# ls /sys/class/uio/uio0/maps/map0/

addr  name  offset  size

root@t2080rdb:~# ls /sys/class/uio/uio0/

dev  device  event  maps  name  power  subsystem  uevent    version

root@t2080rdb:~# ls /sys/class/uio/uio0/maps/

map0

root@t2080rdb:~# ls /sys/class/uio/uio0/maps/map0/

addr  name  offset  size

root@t2080rdb:/sys/class/dma# dmesg | grep fsl-of-dma

[    1.554241] fsl-of-dma ffe100300.dma: dma channel dma-uio0-0 initialized

[    1.559749] fsl-of-dma ffe100300.dma: dma channel dma-uio0-1 initialized

[    1.578141] fsl-of-dma ffe100300.dma: dma channel dma-uio0-2 initialized

[    1.583643] fsl-of-dma ffe100300.dma: dma channel dma-uio0-3 initialized

[    1.589147] fsl-of-dma ffe100300.dma: dma channel dma-uio0-4 initialized

[    1.594657] fsl-of-dma ffe100300.dma: dma channel dma-uio0-5 initialized

[    1.600162] fsl-of-dma ffe100300.dma: dma channel dma-uio0-6 initialized

[    1.605667] fsl-of-dma ffe100300.dma: dma channel dma-uio0-7 initialized

[    1.611206] fsl-of-dma ffe101300.dma: dma channel dma-uio1-0 initialized

[    1.616709] fsl-of-dma ffe101300.dma: dma channel dma-uio1-1 initialized

[    1.622214] fsl-of-dma ffe101300.dma: dma channel dma-uio1-2 initialized

[    1.627726] fsl-of-dma ffe101300.dma: dma channel dma-uio1-3 initialized

[    1.633234] fsl-of-dma ffe101300.dma: dma channel dma-uio1-4 initialized

[    1.638736] fsl-of-dma ffe101300.dma: dma channel dma-uio1-5 initialized

[    1.644240] fsl-of-dma ffe101300.dma: dma channel dma-uio1-6 initialized

[    1.649741] fsl-of-dma ffe101300.dma: dma channel dma-uio1-7 initialized

[    1.655280] fsl-of-dma ffe102300.dma: dma channel dma-uio2-0 initialized

[    1.660783] fsl-of-dma ffe102300.dma: dma channel dma-uio2-1 initialized

[    1.666294] fsl-of-dma ffe102300.dma: dma channel dma-uio2-2 initialized

[    1.671796] fsl-of-dma ffe102300.dma: dma channel dma-uio2-3 initialized

[    1.677300] fsl-of-dma ffe102300.dma: dma channel dma-uio2-4 initialized

[    1.682802] fsl-of-dma ffe102300.dma: dma channel dma-uio2-5 initialized

[    1.688306] fsl-of-dma ffe102300.dma: dma channel dma-uio2-6 initialized

[    1.693809] fsl-of-dma ffe102300.dma: dma channel dma-uio2-7 initialized

root@t2080rdb:/sys/class/dma# uname -a

Linux t2080rdb 4.14.67+gf2f35bbfc16b #1 SMP Tue Jun 4 09:40:37 UTC 2019 ppc64 ppc64 ppc64 GNU/Linux

root@t2080rdb:/sys/class/dma#

Thanks,

Yiping

[bradley_gamble]

Hi yipingwang‌,

Thank you for your support. I can confirm your suggestion has exposed the "/dev/uio*" paths as required.

The Freescale userspace driver code I am using also requires access to the path "/dev/fsl-usdpaa". This appears to come from the kernel option `FSL_USDPAA` in the Freescale kernel. This driver does not appear to be simple to copy across to the manline kernel.

I have cloned the `usdpaa` repo from Freescale Git, however this code is identical to the userspace code that I am currently using in my application. Is it possible to enable the `fs-usdpaa` driver on the mainline kernel, or have it been superseded by a different interface?

Kind regards,

[yipingwang]

USDPAA is not supported in the recent released upstream Kernel.

Kernel configuration options to enable upstream version of DPAA driver for PowerPC as the following.

     CONFIG_FSL_DPAA,  CONFIG_FSL_FMAN and CONFIG_FSL_DPAA_ETH need to be enabled in kernel menuconfig as below:

     Device Drivers  --->

        SOC (System On Chip) specific Drivers  --->

            [*] Freescale DPAA 1.x support  --->

     Device Drivers  --->

        [*] Network device support  --->

        [*]   Ethernet driver support  --->

              <*>     FMan support

              <*>       DPAA Ethernet  ----

Thanks,

Yiping