pcie hot-plug for ls1043a

[xvzhaobin]

I have a custom ls1012a board .There is an FPGA on the board.ls1012a access FPGA by pcie.I load FPGA in linux user space.So I must hot plug pcie.I enable linux hot-plug config an cat 1 > /sys/bus/pci/rescan after load FPGA.I can't find pcie device by lspci. Then I only reboot cpu,I can find pcie device by lspci.

I have a same board for ls1043 an FPGA  .on ls1043a board , I can find pcie device by cat 1 > /sys/bus/pci/rescan.

I look at RM of ls1043a and ls1012a.the parts about pcie are same.

So  why pcie of ls1012a can't hot plug?

if it can't hot plug ,How can I hot plug  pcie device ?

[ufedor]

Which Layerscape SDK version is used?

Can you try with the latest one?

[xvzhaobin]

I use LSDK 19.03

[ufedor]

Please try 2004

[xvzhaobin]

2004 can't work too

[ufedor]

>> It seems that the LSDK issue was not fixed.

It is not a LSDK issue as the test was manual on u-boot to create CTO (Completion Timeout) condition on LS1012A and LS1046A and observed that SoCs behaved differently. However, CTO is not expected in healthy PCIe interface. If CTO happens then we need to see the cause of the CTO instead.

>> There is an FPGA on the board.ls1012a access FPGA by pcie.

>> I load FPGA in linux user space. So I must hot plug pcie.

PCIe base spec states two rules:

1. A component must enter the LTSSM Detect state within 20 ms of the end of the Fundamental reset.
2. A system must guarantee that all components intended to be software visible at boot time are ready to receive Configuration Requests within 100 ms of the end of Conventional Reset at the Root Complex.

These statements mean the FPGA must be configured within a certain finite time, and not meeting these requirements could cause problems with link training and device recognition.

In other words, although we understand customer’s need to load FPGA image, such usage scenario is NOT supported by the native Linux open source Kernel.  It’s also an unsupported scenario by PCIe base spec.

However, the described scenario is not new to industry. Some of the implementation can be referenced from here (please pass-on to customer):

https://www.xilinx.com/support/answers/37406.html

https://www.xilinx.com/support/answers/34777.html

https://www.xilinx.com/support/answers/34800.html

Essentially, it is most important to enumerate only when PCIe link is stable and ready to accept configuration cycles such that there is no UR completion from the EP or Completin Timeout (which is worst case).

Currently, your LS1012A implementation may be passing as the system may be meeting above conditions of stable PCIe link before enumeration or rescan.

If the implementation is correct as discussed above, there should not be an issue with either LS1012A and LS1046A.

[xvzhaobin]

I  don't  think this aboved is right.If I don't load pcie host driver and then load FPGA image, the LTSSM state is L0.  I load pcie driver and don't  load FPGA image, the LTSSM state is detect ative. So I think It seems that the LSDK or ls1012a issue was not fixed .

[ufedor]

>> I load pcie driver and don't  load FPGA image, the LTSSM state is detect alive.

[] - If FPGA image is not loaded then I do not think FPGA PHY is ready to negotiate with LS1012A. LTSSM state machine will not progress in that case.

[ufedor]

sys/bus/pci/rescan essentially, enumerates PCIe bus. There would not be any different in the enumeration process for LS1012A and LS1043A.

In addition to my previous response, I think, customer may be missing to remove the device from driver using the following command before re-configuring the FPGA image or first time configuring FPGA image.

# /sys/bus/pci/<your PCI bus number>/remove

The command would detach the device(s) from driver. Then rescan command can then be run after reloading the FPGA image to re-enumerate FPGA-EP.