PCIe rescan in Linux

[jeffreycoffman]

Hello,

We are developing an FPGA device, which is going to be connected to an i.MX6 as a PCIe endpoint device.  I'd like to configure the FPGA once the system is up and running, and of course at this point the kernel has already probed the PCIe bus and hasn't see anything yet.  I'd like to be able to rescan the PCIe bus once the device is configured and the PCIe core is running.

I've done some searching and seen some reference to doing an echo 1 > /sys/bus/pci/rescan.  This doesn't really have any effect and I am told you need hotplug support for this to work.  So I am wondering the following:

1.  Does 3.0.35 Linux kernel support hotplug for the i.MX6?  I don't see it as option in the menuconfig.

2.  Does my device need to support hotplug, I assume it has to assert PERST#.

3.  My device isn't going to be hot swappable, I just need to delay the Kernels initial probing of the PCIe bus until the FPGA is configured.  Is this possible todo without hotplug, or are pretty much stuck needing the hotplug service?

If someone out there can help explain this in more detail, that would be great.

Thanks

[randysilagi]

Hi Jeffery,

Back in 2013 you were having a problem with PICe rescan.  I'm wondering how you loaded the FPGA.  Did you use one of the SPI channels or GPIO pins?  I'm working on a new product based on the i.MX6sx and we are planning on the loading the FPGA using ecspi0 and then doing the rescan.

Randy S.

[jeffrey_coffman]

Hi,

We load the FPGA using ecspi2.  We never ended needing to use rescan, for our application.

[hendriklugtenbu]

Hi,

These forum posts are all more then a year old. I am wondering if in newer Linux kernels these problems are solved. I found these commits to the kernel:

https://github.com/torvalds/linux/commits/master/drivers/pci/host/pci-imx6.c

History for drivers/pci/host - torvalds/linux · GitHub

I could not find literally that these commits resolve the rescan issues. Has anyone experience with this?

Thanks!

-Hendrik

[richard_zhu]

As I known that the hot-plug is not supported by imx6 pcie.

[jeffreycoffman]

Thanks again!  I will give this a try on my system.

Jeff

[kitz36]

I had a similar issue.  We wanted to reload an FPGA, which is our PCIe endpoint.  This would effectively take the link down.  Bringing the link back up turned out to be a bit of a problem.  The solution was two-fold.

First, I sort of had to hack the Linux kernel to force a true "rescan".  Originally the kernel wouldn't truly probe the endpoint on a rescan.  I ended up forcing the kernel to "forget" the bus by adding a call to "pci_remove_bus()" in drivers/pci/probe.c :: pci_rescan_bus().  See attached.

Second, I had to do another hack to re-initialize the i.MX6 PCIe hardware.  I described that in an earlier post. 

-Charlie

[jeffreycoffman]

Charles,

Thanks for providing your solution.  I was hoping there was a less invasive way of doing this via the hot-plug mechanism, but I still haven't figured out whats involved with that approach.

I assume you made the changes to the re-initialize the i.MX6 PCIe hardware in imx_pcie_pltfm_reinit?

Thanks Again,

Jeff

[kitz36 · Accepted solution]

That is correct.  I added the re-initialization code to my arch/arm/mach-mx6/pcie.c module.  Actually, my pcie.c module has quite a few updates versus the 1.1.0 BSP.  See attached.  Let me know if you have questions.

-Charlie

[gavinli]

We found out it use different way to configure PCIe: It configures RC to use clock from EP

As we use RC clock to drive EP, so I disabled the PHY_MPLL override code in pcie.c

#define ZEP_DEBUG_DONT_INIT_PCI 1
#undef ZEP_DEBUG_DONT_INIT_PCI

#define ZEP_PCIE_OVERRIDE_PHY_MPLL 1
#undef ZEP_PCIE_OVERRIDE_PHY_MPLL     /* !!!!! */

#define IMX_PCIE_CORE_DEBUG
#undef  IMX_PCIE_CORE_DEBUG

#ifdef  IMX_PCIE_CORE_DEBUG
#define IMX_PCIE_PL_DEBUG_BUFFER_SZ (1024)

I did “echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/remove” ,  then I recycled the EP power, and did “echo 1 > /sys/bus/pci/rescan”
The result is Linux kernel crash:

Unhandled fault: imprecise external abort (0x1406) at 0x000c4000
Internal error: : 1406 [#1] PREEMPT SMP
Modules linked in:
CPU: 0    Not tainted  (3.0.35-2508-g54750ff #77)

It also has lots of changed in the pcie.c comparing to my 4.0.0 release.

Would you please help us find if there is a solution for the 4.0.0 release?

Thanks.