Cannot DMA with Hypervisor on T1042

[paterb]

Hello,

I am trying to DMA from a custom ASIC over PCIe and cannot get it to work.  If I take the Hypervisor out of the equation (by just booting Linux), the DMA seems to work fine.  When I add in the Hypervisor, I get a "Transmit Error" from the Freescale ELO driver (fsldma.c). 

fsl-elo-dma ffe100300.dma0: chan0: Transfer Error!

fsl-elo-dma ffe100300.dma0: chan0: irq: unhandled sr 0x00000080

According to the t1042 reference manual, the cause of this error could be a few different things:

On a transfer error (for example, non-correctable ECC errors on memory accesses, parity

errors on flash controller, address mapping errors, and so on), the DMA halts by setting

SRn[TE] and generates an interrupt if MRn[EIE] is set.

Unfortunately, I don't see any method to retrieve more information on what is actually the issue.  Hypervisor prints out no errors or warnings when this happens.  I've read through the Infocenter information on the Hypervisor, and believe I've setup everything correctly.  I have a DMA window setup for the RAM area I would like to DMA into (I also tried setting up a DMA window at the address of the ASIC in PCIe space where I am DMAing from, but that also did not help).

I can forward on my Hypervisor DTS file, the Linux DTS file and also a dump of the Guest device tree if that would help.  I'm really just looking for any clues of where my issue might be or things I could look for to help debug the problem.

Thanks.

Pat

[scottwood]

In order for devices to do DMA, they need to have their liodn configured with a mapping in the hv dts file.  Could you attach yours?

[paterb]

I've attached my HV DTS (hv_t1042rdb_CoreNet.dts), Linux DTS and a dump of Core 0 guest DTS.

Thanks.

Pat

[paterb]

Scott,

The last HV DTS file I attached had a change in the DMA sub windows that I made trying to figure this issue out.  I've reverted those changes and here is the actual DTS I started with.

Thanks for your help.

Pat

[scottwood]

That dts has comments in it, which means it's not a disassembled dtb -- yet it has phandle and linux,phandle properties.  Remove them and replace references to those phandles with label references.  Phandles should be generated by dtc.

As for the DMA problem, you said you were trying to DMA to a PCIe device, but you don't have a mapping for that in the dma-window.  You'll need to create a subwindow for the MMIO (assuming it's that and not PIO) window of the controller you want to access.  While the hypervisor wasn't meant to handle DMA mappings to anything other than memory or PCIe MSIs, you may be able to get it to work by creating a PMA that covers the MMIO region, pointing a subwindow at it, and adjusting the IOVA you program into the DMA engine based on the subwindow address (due to PAMU constraints you probably won't be able to get it to be the same as the physical address).

[paterb]

Scott,

I've updated my DTS file to remove phandle references and use label references instead.

For the DMA issue, we are trying to DMA from the PCIe device into DDR memory (not to the PCIe device).  So do I need to create a dma subwindow for the both the source (PCIe address) and the destination (DDR)?  If that is the case, I think that will be an issue right now because PCIe4 gets mapped to address 0xC 3000 0000 (36 bit address) while DDR is mapped at 0.  Trying to create a DMA window with sub windows to cover that much address space seems impossible due to the constraints of 1 DMA window per device.

Thanks.

Pat

[scottwood]

It doesn't matter what direction you're moving the data.  Both the source and the destination must be mapped.  You can't have an IOVA of 0xc_3000_0000, but you can have a different IOVA that points to a physical address of 0xc_3000_0000.  This will require the software programming the DMA controller to be aware that the IOVA does not match the physical address.  Unfortunately it looks like the hypervisor doesn't allow this to be configured via the device tree, except for special cases like pcie-msi-subwindow.  You'd need to modify setup_subwins() in src/pamu.c to accept a new "guest-phys-addr" or similar property, so the IOVA (guest-addr) can be specified separately from the guest physical address.

[paterb]

Scott,

Thanks again for all the help.  I've started trying to change the HV to add a way to get the IOVA that points to a physical address at 0xc_3000_0000.  Before I get too far along, I thought I would double check that I can get DMA working at all.  So I changed my test code to DMA from DDR to DDR (allocated 2 buffers in Linux with GFP_DMA attribute).  When I attempt to DMA from 1 of these buffers to another one (both are in DDR so they should be covered by my DMA sub-window), I get the same TX Error from the ELO driver in Linux.

fsl-elo-dma ffe100300.dma0: chan0: Transfer Error!

fsl-elo-dma ffe100300.dma0: chan0: irq: unhandled sr 0x00000080

This should be possible correct?  I understand you would never actually DMA from RAM to RAM in the same partition, but the test should still work shouldn't it?

Pat

[paterb]

Scott,

Just wanted to also make sure this is what you were explaining in your last post:

We will need to modify the hypervisor code to support something like this (maybe something similar to the pcie-msi-subwindow code there is now?

Pat

[scottwood]

I was talking about IOVAs (I/O virtual address, i.e. the address you program in the device where the physical address would go in the absence of an IOMMU), not CPU virtual addresses.  I also would recommend choosing an IOVA that doesn't overlap DDR (unless the DDR at 0xc000_0000 isn't assigned to this partition).

It's like the pcie-msi-subwindow code in that it overrides the guest RPN, but I'd recommend something that is just a straightforward guest-phys-addr property as I previously described, rather than a boolean option that gets an RPN via magic.