PCIe BAR Configuration Failed When Interfaced with Freescale i.MX6Q

[sushantmahajan]

Hello,

We are trying to validate PCIe Communication with Lattice ECP5 to Freescale.iMX6 Device.

During enumeration we have seen that the EP is getting detected but the BAR Configuration remains half-finished hence BARs inside EP remain in their initial state and hence driver doesn’t get loaded completely into the Kernel Module.

For successful BAR Configuration ARM RC should program upper bits of BART with the starting address range assigned for the interfaced PCIe EP Card.

Could you please assist us in this if you have solution for the above problem or atleast the workaround which will solve the above problem?

Let us know if you need more details on this.

Thanks,

Message was edited by: Sushant Mahajan

Original Attachment has been moved to: lscpcie2_Linux26.zip

[sushantmahajan]

Lattice EP is requesting 256 KB of Memory to Freescale ARM.

We even tried to reduce this to 4KB but still we observe the same problem.

Let us know if you need further information.

Thanks,

[jamesbone]

We are checking your issue internally as soon as we have an update we will back to you.

[sushantmahajan]

Hello,

Any update on this?

The problem is when “pci_request_regions” API is called in the Driver at #1137.

Driver Source Code is attached.

File Name is lscpcie2.c

If we comment this line then we are able to insert the Driver into the Kernel Module without any errors.

But we cannot comment this since it should mark the PCI region associated with PCI device device pdev BAR bar.

This call is not getting returned successfully (giving an error) because of which the address inside the PCI regions are not accessible.

On Successful return this should mark the owner of the region(EP Driver in this case) and should return (physical) address and length.

This happens only when we interface Versa with ARM.

Appreciate your help in this.

Regards,

[b47504]

From your log, we saw the return value is EBUSY. So the region should have been requested by other driver module.

I think you can figure out if someone is using this or not released when exited in your system, firstly.

[sushantmahajan]

Yuan,

This is the typical reason for this error.

Unfortunately this is not the cause since we don’t have any other driver inserted who has requested this region previously.

That is why we are wondering why this error is coming even though this has not been getting utilized by other module? Any other possible scenario where we can focus on?

Is it possible that this is an issue or some sort of limitation with the Kernel version or the OS running on i..mX6?

Appreciate your help in this.

Thanks,

Sushant

[b47504]

Imx6 pcie is used as RC  in your case, right?

The issue is special for “Lattice EP”, or is the same for other ep devices?

Best Regards,

[sushantmahajan]

Yuan,

Any update on this?

I also tried to reduce the memory request just to 256 bytes but still getting below error.

BAR 1: can't reserve

It appears that RC has already assigned the memory to other PCI Peripherals.

Lspci Console gives me below output.

root@iWave-G15M /boot$ lspci

00:00.0 Class 0604: 16c3:abcd

01:00.0 Class 0604: 10b5:8505

02:01.0 Class 0604: 10b5:8505

02:02.0 Class 0604: 10b5:8505

02:03.0 Class 0604: 10b5:8505

02:04.0 Class 0604: 10b5:8505

03:00.0 Class 0000: 1204:ec30

Device with 1204:ec30 is Lattice EP.

Can we force RC to disable other devices so that atleast RC should assign Memory to Lattice EP and verify using lspci?

Regards,

Sushant

From: Sushant Mahajan

Sent: Friday, August 08, 2014 2:46 PM

To: 'jive-1992769512-4v0e-2-94kz@freescale.hosted.jivesoftware.com'

Subject: RE: - PCIe BAR Configuration Failed When Interfaced with Freescale i.MX6Q

Yuan,

Yes Imx6 is used as RC.

This is special to Lattice EP Devices.

Thanks,

Sushant

[b47504]

Yes, as you said, it seems that it is special for Lattice EP.

I cannot reproduce the issue used intel wifi card and your driver.

Could you please help on:

(1) “Echo 7 > /proc/sys/kernel/printk” and “dmesg” and capture more message.

(2) Please try with the following modification:

--- a/drivers/pci/pci.c

+++ b/drivers/pci/pci.c

@@ -2392,7 +2392,7 @@ static int __pci_request_region(struct pci_dev *pdev, int bar, const char *res_n

{

struct pci_devres *dr;

- if (pci_resource_len(pdev, bar) == 0)

+ if (pci_resource_len(pdev, bar) == 0 || pci_resource_flags(pdev, bar) == 0)

return 0;

[sushantmahajan]

Hi Yuan,

Thanks for your email.

At EP Side we have 32 bit BAR setting :

We have set 32'hfffcffff as intial power on reset contents which remains as is but actually this should be modified by RC with the starting physical address.

I have also tried to reduce the memory size in the BAR setting but it didn't help.

Anything specific that we should check on EP side?

Could you elaborate "For EP side c=0,1,2,5"??

Since pci_request_regions is giving error we have also tried to hard code the BAR value from driver and upon reading we are able to retrieve the value with which BARs are written.

Let me know your views in this.

Thanks,

Sushant

[b47504]

In general, for ep, can have 6 bars. For 0….5, I just want to see every bar’s setting.

For base address register, the initial value indicate the bar type:

Bit 0: 0 – memory bar 1 – I/o bar

Bit 2:1 00 – 32bit bar 10 – 64bit bar

Could you please try with the rules?

And how you set the size your request?

Best Regards,

[sushantmahajan]

Hi Yuan,

We have set BAR0 and BAR1 with 32'hfffc0000 meaning that :

Bit 0 :- 0- Memory

Bit 2:1 :- 00 set for 32 bit BAR.

We have set the upper 31:20 bits with all 1's and tied the LSB to 0 to indicate the Request size.

So if you will see bit 18 is set to 1 so 2^18 = 256 KB.

Specifically what you want me to try?

We have also tried to request for I/O Memory and reducing the memory size but the same error persists.

Thanks,

Sushant

[b47504]

For rc, it will write ~0(ffffffff) to base address register, then re-read the base address register to get the request size.

Could you please help on witing ~0 to base address register, and then read the value from the base address register on your side?

Best Regards,

[sushantmahajan]

Yuan,

We have tried to set BARs with all 0’s and the error doesn’t come but this will not help as per Protocol point of View.

I am still not sure what exactly you want us to check? Can you elaborate?

Regards,

Sushant

[b47504]

Write ~0 to base address register, then read out the register value. The value is the size you request to RC by bar.

Could you please make sure when rc write ~0 to your ep bar register ,then rc read the bar register, the rc can get the right size we expected.

Best Regards,

[sushantmahajan]

Yuan,

Writing all 0’s to EP BAR will request 0 bytes of memory to RC.

So reading this value will be all 0’s?

I am not sure if RC will program the requested memory size since during decoding it will decode the request as 0 bytes?

I am still not clear what exactly you mean? Can you elaborate?

-Sushant

[b47504]

It is ~0(ie 0xffff ffff) ,rather than 0.

According to pcie standard protocol, the RC will write 0xffff ffff to bar register, and read from the bar register, now the value return from the register should be the size it requested.

After RC get the size, rc will recover the value you set before.

Best Regards,

[jpa]

Was there a resolution to this problem? 

I believe we're facing a similar problem with the imx6 and a Lattice ECP3 FPGA.   The Lattice FPGA design has both BAR[0] and BAR[1] at 0xFFFC 0000, and it does not get re-mapped by the imx6 pcie system. 

The same FPGA firmware in a QorIQ P1010 ends up mapped to 0x80000000 and 0x80000000+256k before our FPGA driver ever gets involved, I think. 

pci_request_regions reserves bar[0], but fails on bar[1], presumably because it's at the same address. 

We're using 3.14.28.

John

[jpa]

In case it helps someone that follows...in our case, the problem boiled down to the FPGA having it's class set to "0" in the FPGA design.  The FPGA design software made it appear to be non-zero, but when Linux read the value, it was zero.  This gets interpreted as "class unknown" and the pci kernel software ignores the device and does not assign resources to the BAR.  Fudging the class to a non-zero value fixed the behavior.  

This behavior appeared to be specific to the imx6...we used the same FPGA with other processor builds without this problem.  But the section of code that it was in did not, in fact, appear to be machine specific, so I'm guessing that it's a side effect of a newer kernel, and not specific to the imx6. 

John

[sushantmahajan]

Hi Yuan,

I am attaching the output of dmesg once again.

Also we have tried to update the pci.c file as per your comments and we have recompiled the Kernel Image.

With the above Test we can load the driver into the Kernel Module without an error but I believe we are just masking the actual error and I still not have the Clarity that i.mX6 RC is actually assigning the Physcial addresses for Lattice Device.

If it is assigning then I would like to verify the actual address assigned to Lattice Device.

Could you help us at least with the possible reasons for this unexpected behavior?

Doing Dmesg I can see that address range is getting assigned but the problem is coming during programming of physical address into the configuration space of EP.

Anything that we can check or modify on the EP Side?

Regards,

Sushant

From: Sushant Mahajan

Sent: Tuesday, August 12, 2014 7:14 PM

To: 'jive-1168543470-4v0e-2-956s@freescale.hosted.jivesoftware.com'; 'Yuan.Zhao@freescale.com'

Cc: James Kenney (Jim.Kenney@freescale.com); Akbar Matin

Subject: RE: - PCIe BAR Configuration Failed When Interfaced with Freescale i.MX6Q

HI Yuan,

Attached is the output of dmesg.

We are working on changes in pci.c file and recompiling the Kernel.

Regards,

Sushant

[b47504]

“Flag == 0” indicate something wrong  in the pcie device probe process.

Yes, it will be more helpful if we can check bars setting on the ep side-

For ep side(x = 0, 1, 2….5):

(1)          Read bar[x]                                           (Help capture the value read from the bar)

               ->

(2)          Write 0xffff ffff (~0) to bar[x]

                            ->

(3)          Read bar[x]                                          (help capture the value read from the bar, after write ~0)