How to use the rmu application (from usdpaa-apps) on the P4080?

[andreloddenkemp]

Hello everybody,

I have a P4080 running with a BSP created from the SDK version 1.8. So far it runs fine, I can use the rapidIO kernel driver to enumerate my system and I can use the SRA application (from usdpaa-apps) to use NWRITE and NREAD from user space.

Now I wanted to use the rmu application (from usdpaa-apps), which is supposed to handle type11 packets and doorbells. For that I compiled the kernel module fsl_rmu_uio.ko and I compiled the rmu application. I can load the module (insmod fsl_rmu_uio.ko) and I can start the application (./rmu). However, when the application tries to initialize the RMU message units or the doorbell unit, it crashes with a segmentation fault:
*******************************************************
[21453.596549] rmu[1664]: unhandled signal 11 at ffffffff nip 10003e08 lr 10003df8 code 30001
*******************************************************

This segmentation fault occurs after the application tries to get shared memory from the kernel module, using the mmap call in function fsl_rmu_unit_uio_init() in file rmu_driver.c. This mmap call returns with MAP_FAILED (-1), which weirdly is not caught by the application, because it only checks the return value for 0, but not for -1 (MAP_FAILED). Thus it tries to access the address 0xffffffff afterwards and crashes with segmentation fault.

I looked into the responsible kernel module (fsl_rmu_uio.c) and there I can see that it does not have any function to handle the mmap call. So it is not surprising to me that it returns with MAP_FAILED, I think. I am just wondering how that can be. How is the application supposed to run? It seems like it could never work as it is. What am I misunderstanding? What am I missing?

So ...

1.) Is there anyone with experience regarding usdpaa, usdpaa-apps and specifically the rmu application, who can help me with my issue?

2.) Or are there other ways of using type11 messages on P4080?

3.) Other ways of receiving doorbells on P4080?

p.s. All the source code is available here (http://git.freescale.com/git/cgit.cgi/ppc/sdk/usdpaa.git/) and here (http://git.freescale.com/git/cgit.cgi/ppc/sdk/usdpaa/usdpaa-apps.git/).

[andreloddenkemp · Accepted solution]

I solved my problem in the meantime.

The segmentation fault was caused by accessing a memory location with the address 0xffffffff. This happened, when the rmu driver (rmu_driver.c) tries to mmap some CPU registers. The mmap fails and returns (correctly) with -1 (MAP_FAILED), which the driver fails to notice. The driver only checks for 0/NULL, but not for MAP_FAILED/-1. So he proceeds and tries to access the memory at -1/0xffffffff.

So I checked the mmap mechanic behind the rmu_driver.c, which can be found in the kernel source file uio.c, which is the generic userspace IO driver source code. Here the mmap of the rmu_driver.c arrives, because the rmu kernel module fsl_rmu_uio.c does not have its own mmap implementation. The uio_mmap function in uio.c then checks a few things, like comparing the requested memory size vs the allowed memory size of the rmu unit (specifically the messe units 0, 1 and the doorbell unit). Here the mmap fails, because the allowed mem size is set to 0x100 (for the rmu message units) and 0x80 (for the doorbell unit), but a call to mmap will always try to get AT LEAST one page (4096 bytes). So 4096 is bigger than 0x100 or 0x80, which is why the mmap fails, which is why it returns with MAP_FAILED.

The mem sizes (0x100 and 0x80) are set via the device tree binding for the rmu unit, more specifically the "reg" parameter in the message units and the doorbell unit. So increasing the size there to 0x1000 is one option to let the mmap call succeed, but I am not sure if that is wise. There might be other kernel/driver components who read out the device tree parameters and use those sizes for other things. I am not sure about the consequences of this possible solution.

My personal and current solution is to add a mmap function to the fsl_rmu_uio.c kernel module, where I do not do this kind of check. It works fine and I can now send and receive doorbells between two P4080 boards. However I am not sure if this is an optimal solution.

I want to investigate some more and see if there are more elegant solutions, but it works now and I can use the rmu application as well as the fsl_rmu_uio.ko kernel module just fine.

Any input from you guys is appreciated. Maybe this helps someone else as well.

Best regards,

Andre

[yang_lei]

Hi Andre,

I am facing the same problem, and I tried the first method you mentioned,  to increase the size to 0x1000. And RMU application can run now, but I got those error excution prints:

rmu> rmu -op dbell ar 1
OP: doorbell rx thread ...create success!
rmu> --doorbell1: sid 00 tid 00 info 0000
--doorbell2: sid 00 tid 00 info 0000
--doorbell3: sid 00 tid 00 info 0000
--doorbell4: sid 00 tid 00 info 0000
--doorbell5: sid 00 tid 00 info 0000

So, I think you are right, it is not wise to incresing the size.

Now I want to try your current solution:  add a mmap function to fsl_rmu_uio.c kernel module. But I don't know how to overwrite the function. Could you please paste your fsl_rmu_uio.c file here?

Hope to get your help soon and sorry for my poor English.

Best Regards,

Lei

[andreloddenkemp]

Hi Lei,

here is all you need, it is only a small patch to apply:

1. Adding the mm.h header file.

2. Adding the rmu_uio_mmap() function, containing the actual mmap mechanisms in it.

3. Making sure that the newly added function is called, when an application calls for "mmap". We do this by assigning our new function to the file operations of the uio device, when the device is set up. We assign our new rmu_uio_mmap() function to the file operation "mmap".

diff -burN linux-3.12-r0/drivers/uio/fsl_rmu_uio.c linux-3.12-r0-srio/drivers/uio/fsl_rmu_uio.c
--- linux-3.12-r0/drivers/uio/fsl_rmu_uio.c     2017-06-26 13:44:00.651883090 +0200
+++ linux-3.12-r0/drivers/uio/fsl_rmu_uio.c     2017-06-26 13:41:16.788991528 +0200
@@ -21,6 +21,7 @@
 #include <linux/uio_driver.h>
 #include <linux/list.h>
 #include <linux/io.h>
+#include <linux/mm.h>
 
 /* rmu unit ID, based on the unit register off-set */
 #define RMU_UNIT_MSG0 0
@@ -74,6 +75,29 @@
      return 0;
 }
 
+static int rmu_uio_mmap(struct uio_info *info, struct vm_area_struct *vma)
+{
+     int mi;
+     struct uio_mem *mem;
+
+     if (vma->vm_pgoff < MAX_UIO_MAPS) {
+          if (info->mem[vma->vm_pgoff].size == 0)
+               return -EINVAL;
+          mi = (int)vma->vm_pgoff;
+     } else {
+          return -EINVAL;
+     }
+     mem = info->mem + mi;
+
+     vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+     return remap_pfn_range(vma,
+                      vma->vm_start,
+                      mem->addr >> PAGE_SHIFT,
+                      vma->vm_end - vma->vm_start,
+                      vma->vm_page_prot);
+}
+
 static int __init rmu_unit_uio_setup(struct rmu_unit *unit)
 {
      int ret;
@@ -100,6 +124,7 @@
      info->uio.mem[0].size = resource_size(unit->res);
      info->uio.mem[0].internal_addr = unit->regs_win;
      info->uio.mem[0].memtype = UIO_MEM_PHYS;
+     info->uio.mmap = rmu_uio_mmap;
      info->uio.open = rmu_uio_open;
      info->uio.release = rmu_uio_release;
      info->uio.priv = unit;


‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

If you have any more questions or anything is unclear, feel free to ask.

Best regards

Andre

[yang_lei]

Hi Andre,

Have you add any irq handling code in fsl_rmu_uio.c ? I checked the fsl_rmu_uio.c file, there is a irq member in rmu_unit struct:

struct rmu_unit {

...

int irq;

...

}

but there is no operations on this member at all. The irqs of rmu node in dts(60 - 63 for msg unit and 56, 57 for dbell unit) are not registered at all.

And I checked the fsl_rmu.c file located in /arch/powerpc/sysdev/ and find there are some聽 irqreturn_t functions used for update rmu msg/debll in queue regs when irq received. Those functions should be implemented in fsl_rmu_uio.c too unless the fsl_rmu.c file is also included to the kernel building tree.

But the UIO_FSL_RMU configuration is depending on RAPIDIO=n, which means that if I include the fsl_rmu.c file in building tree, the fsl_rmu_uio.c can not be included. 聽

I'd like to know how did you solve the problem. Or, if you have some suggestions, please let me know.

Thanks:)

:smileyhappy:

Regards,

Lei

[yang_lei]

Hi Andre，

Thanks for your quick reply.

I applied the modification in your patch, and it works（mmap issue is solved）. But the wrong prints are still there, I receive doorbells like this continuously if I create a doorbell receive thread：

--doorbell2: sid 00 tid 00 info 0000

I am using the dtb and rootfs built by myself, maybe that is why I continuously got those doorbell that should not be received:

      I only added rmu node to "p2041rdb.dts" and built my dtb file, maybe there are other essential "usdpaa dts nodes" which are  needed.

      And in the rootfs, I am using "ln -s /dev/uioXX /dev/rmu-uio-doorbell" to create rmu-uio-doorbell device for RMU application to use. Maybe the rootfs in ths SDK add those rmu-uio-xx devices in a different way.

Now I am tring to replace dtb and rootfs with those in the SDK package.

And thanks again for your help.

Best Regards,

Lei

[yipingwang]

Hello Andre Loddenkemper,

Please try to use "RIONET" in Linux Kernel to verify RMU driver on P4080DS.

Please refer to "7.23.1 RapidIO User Manual" in SDK 1.8 document "QorIQ-SDK-1.8-IC-RevA.pdf" for details.

Have a great day,
TIC

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[andreloddenkemp]

Hello Yiping Wang,

thank you for your answer!

I have an additional question: Can I use RIONET to test doorbell communication with another RapidIO device, which is not using RIONET? To ask more generally: what is the best way of testing doorbell communication?

Best regards,

Andre

[yipingwang]

Hello Andre,

You need to setup RIONET on two target boards. In fact we often use RIONET to test doorbell communication during evaluation or bringing up.

Have a great day,
TIC

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[andreloddenkemp]

Hello Yiping Wang,

please excuse me, I should have been clearer in my previous post.

I have one P4080 and one DSP board. I have full control over the P4080 and no control over the DSP board. I now want to receive doorbell messages from the DSP board. The DSP board is running some kind of rapidio stack, but it is not possible to run RIONET on it, it is something else, which I have no control over.

So my two exact questions are:

1. Can I somehow receive doorbells from the DSP board?

2. Can I somehow send doorbells to the DSP board?

The rmu application sounded fine for this purpose, but unfortunately I can not get it to work, as described in my original post. So now I am in need of an alternative and I can't seem to find one.

Any ideas?

Best regards,

Andre