GFX Ring Test Failure on AMDGPU(E9171)

[Ganesh3955]

I am working on the i.MX95 DevKit (8GB) with AMDGPU (E9171) configured as a PCIe endpoint connected through an x2 interface.
On the i.MX95, there is no 64-bit prefetchable range present by default. However, for this AMDGPU to function, it requires a minimum 8GB 64-bit prefetchable PCIe range.
To address this, I added a custom range in the device tree (imx95-ig61d-prgwd.dts) as shown below:

&pcie0 {
       pinctrl-0 = <&pinctrl_pcie0>;
       pinctrl-names = "default";
      fsl,refclk-pad-mode = <IMX8_PCIE_REFCLK_PAD_OUTPUT>;
      reset-gpio = <&i2c4_gpio_expander_20 6 GPIO_ACTIVE_LOW>;
      vpcie-supply = <&reg_pcie0>;
      status = "ok";
      ranges = <0x81000000 0x0 0x00000000 0x0 0x6ff00000 0 0x00100000>,//IO ports- default
     <0x82000000 0x0 0x00000000 0x9 0x00000000 0 0x10000000>, //32 bit non-prefetchable -        default
     <0xC2000000 0x0 0x10000000 0x9 0x10000000 0x1 0x00000000>; //64 bit prefetchable -             custom added
};

I am able to assign 4GB as the maximum size for the custom range. If I increase it beyond that, the driver gets stuck at 'Chained IB support enabled!'
After increasing the range size to 4GB driver goes further and gfx test fails
I found a similar issue in the forum stating that fsl_pamu should be disabled, but in my case it is already disabled in the kernel 'CONFIG_FSL_PAMU=n'
Reference:
https://community.nxp.com/t5/T-Series/T2080-PCIe-transfer-error-with-PAMU-enabled/m-p/1310923
How can I solve this problem? Please find the test log below.

[Zhiming_Liu]

Hi @Ganesh3955 

1. I confirmed with internal team before: The i.MX95 features two independent PCIe Gen 3 x1 controllers that cannot be aggregated into a single x2 link; your boot logs also clearly show that the negotiation result for this link is “PCIe Gen 3 x1 link up.”

2. According to lspci -vvv, the maximum supported size for BAR0 under Resizable BAR is only 4GB, not 8GB. Your log already states this clearly: BAR 0: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB.

3. As mentioned in the discussions regarding amd-gfx, amdgpu attempts to set the GPU's BAR0 to the largest possible value; if there is insufficient space or the device does not support a larger size, it falls back.

https://lists.freedesktop.org/archives/amd-gfx/2020-December/057339.html

Based on the current logs, it’s not that amdgpu doesn’t want to expand, but rather that the contiguous 64-bit prefetchable space following the RC is insufficient, non-contiguous, or does not meet alignment requirements. The starting address of your current window is 0x10000000 (PCI-side), which is not a 4GB naturally aligned boundary.

Best Regards,
Zhiming

[Ganesh3955]

HI @Zhiming_Liu 

Thank you for quick reply,
Can you suggest me a proper 64-bit prefetchable space with 8GB size for this PCIe BAR

Best Regards,
Ganesh

[Zhiming_Liu]

Hi @Ganesh3955 

Confirmed with PCIE owner, the supported size of BAR0 can only up to 4GB size.

Best Regards,
Zhiming

[Ganesh3955]

Hi @Zhiming_Liu 

Is this limitation is for i.MX95 only or for T2080 processor as well? Please confirm how much of address space can be allocated in T2080 processor.

Best Regards,
Ganesh

[Zhiming_Liu]

Hi @Ganesh3955 

This specifically refers to the i.MX95.

Best Regards,
Zhiming

[Ganesh3955]

Hi @Zhiming_Liu 

Thank you for your reply,

How i can continue my testing with E9171 GPU using i.MX95 board. And why GFX ring test is failing, is there any solution for this?

Regards,
Ganesh