iMX8MP PCIe initialization in U-Boot

[rbeims]

Hello,

A customer is initializing an NVMe drive in U-Boot to load their image (kernel and rootfs) on the iMX8MP. This is working without any problems, the NVMe disk is properly detected, and the system starts booting. However, in some specific SoM's, when the Linux kernel initializes the PCIe interface, the link won't come up. This happens about 30% of the time on affected SoM's.Here's the error message when it fails to initialize:

phy phy-32f00000.pcie-phy.0: phy poweron failed --> -110

Non affected SoM's work 100% of the time. If the PCIe is not initialized in U-Boot, the affected SoM's works 100% of the time on Linux, and we ran 12h stress tests on the NVMe disk without any problems.

The problem is very likely associated to the way the Linux kernel is initializing an already initialized PCIe interface. Because the kernel has no knowledge that the interface was already initialized, it does the entire process from scratch, and this fails in some modules sometimes.

I'm thinking there could be a PLL that doesn't lock properly when it is being reconfigured without being in a reset state. However, as previously noted, this is not reproducible on 100% of the hardwares, so there must also be some kind of HW variation (SoC, SoM) that is also having an effect.

Can anyone give hints about what could be going wrong?

Thanks,

Rafael

[rbeims]

Thanks, after this information, I found that a PHY reset before the linux driver starts solves the issue.

[Bio_TICFSL]

Hello,

I can identify a likely cause for the intermittent failures when Linux kernel initializes an already-initialized PCIe interface.

The problem you're experiencing appears to be related to the PCIe interface initialization sequence between U-Boot and Linux. When U-Boot initializes the PCIe interface to load your image from NVMe, and then Linux attempts to re-initialize the same interface without proper reset, it creates a race condition that fails approximately 30% of the time on affected SoMs.

This is consistent with a PLL configuration issue. When Linux kernel tries to reconfigure an already initialized PCIe interface, it may encounter problems with PLL lock when the hardware isn't properly reset between initializations.

There are three potential solutions:

1. Disable PCIe initialization in U-Boot completely, since you've confirmed this works 100% of the time. This would be the most reliable approach.

2. Implement proper PCIe suspend/resume functionality in Linux by adding sys entries:
- Create a patch to add sys entry for PCIe suspend/resume
- Remove PCIe EP device: `echo 1 > /sys/bus/pci/devices/0000\:00\:00.0/remove`
- Disable PCIe: `echo 1 > /sys/devices/platform/soc\@0/33800000.pcie/pcie_dis`
- Enable PCIe: `echo 0 > /sys/devices/platform/soc\@0/33800000.pcie/pcie_dis`
- Rescan PCIe bus: `echo 1 > /sys/bus/pci/rescan`

3. Build the PCIe driver as a module and modify your boot sequence:
- Boot into Linux first
- Wait for necessary initialization
- Load the PCIe driver module when needed

This issue has been observed on other i.MX8MP implementations, especially with variations in hardware tolerance across different SoM batches. The Linux kernel's PCIe driver assumes it's working with uninitialized hardware, which can cause conflicts when that's not the case.

Regards

[rbeims]

Thank you for the answer!

Our customer wants to load their kernel image and rootfs from the NVMe disk, and this is why they added the U-Boot initialization. Because of this, initializing the PCIe/NVMe after boot is not an option.

I already checked, and they would be fine if there's a way to properly deinitialize the PCIe interface in U-Boot before starting the kernel. I already tried a PCIe reset, but this didn't solve it.

From what you mention, it really looks like the problem is more associated with the PLL's than with the PCIe peripheral. Also, from your suggestions about how to solve it, it almost looks like it could be a PLL that serves more peripherals than just the PCIe.

Would there be a specific PLL we could try to reset before booting into Linux? What would be the correct way to reset it to avoid these problems?

[Bio_TICFSL]

Hi,

The pll need to be checked in the device tree and changes if you need it.

Regards