iMX8MQ: How to configure PCIE1 when using renesas,9fgv0241 for refclks?

[abhishekojhasfl]

Hi All,

We have a iMX8MQ based custom board and we are using a renesas,9fgv0241clock generator to provide REFCLK clock to PCIE1(DIF0) and Wi-Fi(DIF1) module which is connected to PCIE1.

Here is the device tree configuration:

//external osc
clk_xtal25: clock-xtal25 {
compatible = "fixed-clock";
#clock-cells = <0>;
clock-frequency = <25000000>;
};

//Renesss 9FGV0241 clk generator
pcieclk: clk@68 {
compatible = "renesas,9fgv0241";
reg = <0x68>;
clocks = <&clk_xtal25>;
#clock-cells = <1>;
};

&pcie0 {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_pcie0>;
reset-gpio = <&gpio1 9 GPIO_ACTIVE_LOW>;
clocks = <&clk IMX8MQ_CLK_PCIE1_ROOT>,
<&pcieclk 1>,
<&pcieclk 0>,
<&clk IMX8MQ_CLK_PCIE1_AUX>;
vph-supply = <&vgen5_reg>;
status = "okay";
};

dmesg logs:

[ 2.005173] clk-renesas-pcie-9series 1-0068: No cache defaults, reading back from HW
[ 2.251822] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
[ 2.263536] imx6q-pcie 33800000.pcie: IO 0x001ff80000..0x001ff8ffff -> 0x0000000000
[ 2.271745] imx6q-pcie 33800000.pcie: MEM 0x0018000000..0x001fefffff -> 0x0018000000
[ 2.493576] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 4G
[ 3.506430] imx6q-pcie 33800000.pcie: Phy link never came up
[ 4.520275] imx6q-pcie 33800000.pcie: Phy link never came up
[ 4.528952] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
[ 4.608823] pcieport 0000:00:00.0: PME: Signaling with IRQ 206

CCM_TARGET_ROOT [MUX]:
root@m18pisv1-dev:~# devmem 0x3038a400 => PCIE1_CTRL
0x15000007
root@m18pisv1-dev:~# devmem 0x3038a380 => PCIE1_PHY
0x00000000
root@m18pisv1-dev:~# devmem 0x3038a300 => PCIE_AUX
0x11000000

dmesg output shows "Phy link never came up" and Wi-Fi module is not detecting. Please let me know what configurations needs to be done to use the external clk generator.

[joanxie]

what bsp version do you use? did your board design as the same as nxp imx8mq evk board? share your whole dts file with me, and could you measure the reference clock for pcie?

[ExeThun]

Kuchh bhi **bleep** 

[abhishekojhasfl]

Hi

Thanks for quick reply.

I am using latest bsp : LF6.6.3_1.0.0 (Nanbield) and yes our HW design is based on imx8mq evk.

I see proper ~100MHz clk generated from DIF0 and DIF1 pins of clk generator. Also verified it on the Wi-Fi modules side.

One more point:

Only PCIe1 is enabled in device tree, PCIe2 is disabled. But if we enable both in device tree then kernel hangs.

Attaching dmesg logs.

[joanxie]

before I review your SW code, could you confirm your HW design is total fine? I attached the HW design guide, pls check if your HW design is total fine, and did you test other bsp version? like 5.15? do you mind testing on the old bsp?

[abhishekojhasfl]

Hi,

Thanks for sharing the document. It really helped in fixing the PCIe2 hang issue.

• Regarding PCIe1 - Wi-Fi issue.

Only difference I found is we have not connected CLKREQ# processor pin to CLK generator and Wi-Fi module as mentioned in the HW document. Is that pin really required?

From the debugfs output I see that pcie_phy clk is not enabled.

cat /sys/kernel/debug/clk/clk_summary |grep pcie1
pcie1_ctrl 1 1 0 250000000 0 0 50000 Y
pcie1_root_clk 1 1 0 250000000 0 0 50000 Y
pcie1_phy 0 0 0 100000000 0 0 50000 N
pcie1_aux 1 1 0 10000000 0 0 50000 Y
pcie1-refclk 1 1 0 100000000 0 0 50000 Y

I am working on building the 5.15 release, Will provide update soon.

• Regarding PCIe2 hang issue:

We are not using external clk generator on PCIe2 side but pic-imx6.c by default assumes that external clk generator is used so I had to change that for PCIe2 controller and after that PCIe2 comes up without any issue. Still need to test the EP connectivity once below HW issue if fixed.

There was one HW issue on the PCIe2 side that we are fixing REF_CLK pins were not connected to CLK2_P/N (ball T22 and U22) instead PCIEx_REF_PAD_CLK_P/N. we are fixing that.

[joanxie]

if you use 9fgv0241 as imx8mq evk board, pls connect the CLKREQ# signal, otherwise would cause phy link up issue, you can try to connect and test again

[abhishekojhasfl]

Hi

Thanks for all the inputs.

Now we got the new HW with CLKREQ# changes. I am still facing same issue. I also tried using 5.15.71 kernel (imx-linux-kirkstone) but same result probe fails after link up try.

5.15.71 dmesg logs:

root@m18pisv1-dev:~# dmesg |grep pci
[    1.411083] ehci-pci: EHCI PCI platform driver
[    1.427191] ohci-pci: OHCI PCI platform driver
[    1.960100] imx6q-pcie 33800000.pcie: supply epdev_on not found, using dummy regulator
[    1.996885] imx6q-pcie 33800000.pcie: PCIe PLL is locked.
[    2.045774] imx6q-pcie 33800000.pcie: iATU unroll: enabled
[    2.055020] imx6q-pcie 33800000.pcie: Detected iATU regions: 4 outbound, 4 inbound
[    2.066530] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
[    2.081884] imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff -> 0x0000000000
[    2.093835] imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff -> 0x0018000000
[    2.193193] imx6q-pcie 33800000.pcie: Detected iATU regions: 4 outbound, 4 inbound
[    3.583338] imx6q-pcie 33800000.pcie: Phy link never came up
[    3.589553] imx6q-pcie: probe of 33800000.pcie failed with error -110

6.6.3-lts-next dmesg logs:

root@m18pisv1-dev:~# dmesg |grep pcie
[    1.873674] clk-renesas-pcie-9series 1-0068: No cache defaults, reading back from HW
[    2.380739] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
[    2.381148] imx6q-pcie 33c00000.pcie: host bridge /soc@0/pcie@33c00000 ranges:
[    2.388076] imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff -> 0x0000000000
[    2.395298] imx6q-pcie 33c00000.pcie:       IO 0x0027f80000..0x0027f8ffff -> 0x0000000000
[    2.403471] imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff -> 0x0018000000
[    2.411798] imx6q-pcie 33c00000.pcie:      MEM 0x0020000000..0x0027efffff -> 0x0020000000
[    2.428785] imx6q-pcie 33c00000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 4G
[    2.640787] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 4G
[    3.435150] imx6q-pcie 33c00000.pcie: Phy link never came up
[    3.636094] imx6q-pcie 33800000.pcie: Phy link never came up
[    4.448381] imx6q-pcie 33c00000.pcie: Phy link never came up
[    4.456739] imx6q-pcie 33c00000.pcie: PCI host bridge to bus 0001:00
[    4.539463] pcieport 0001:00:00.0: PME: Signaling with IRQ 209
[    4.648560] imx6q-pcie 33800000.pcie: Phy link never came up
[    4.654672] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
[    4.732194] pcieport 0000:00:00.0: PME: Signaling with IRQ 211
root@m18pisv1-dev:~# 

In imx8mq-evk schematic, we can see that same Renesas CLK generator is used but that is not configure in the device tree.

 Here is the pcie device tree nodes. Attaching complete device tree in attachment. Is there any imx8mq based reference dts file available, which is using external clk generator?

	//external osc
	clk_xtal25: clock-xtal25 {
		compatible = "fixed-clock";
		#clock-cells = <0>;
		clock-frequency = <25000000>;
		clock-output-names = "xtal_25M";
	};

&i2c2 {
	//Renesas 9FGV0241 clk generator
	pcieclk: clk@68 {
		compatible = "renesas,9fgv0241";
		reg = <0x68>;
		clocks = <&clk_xtal25>;
		#clock-cells = <1>;
	};
};

&pcie0 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_pcie0>;
	reset-gpio = <&gpio1 9 GPIO_ACTIVE_LOW>;
	wake-gpio = <&gpio1 6 GPIO_ACTIVE_LOW>;
	clocks = <&clk IMX8MQ_CLK_PCIE1_ROOT>,
		 <&pcieclk 1>, // pcie bus
		 <&pcieclk 0>, // pcie phy
		 <&clk IMX8MQ_CLK_PCIE1_AUX>;
	vph-supply = <&vgen5_reg>;
	status = "okay";
};
&iomuxc {
	pinctrl_pcie0: pcie0grp {
		fsl,pins = <
			MX8MQ_IOMUXC_I2C4_SCL_PCIE1_CLKREQ_B		0x76	//CLKREQ
			MX8MQ_IOMUXC_GPIO1_IO09_GPIO1_IO9		0x19	//PERST#
			MX8MQ_IOMUXC_GPIO1_IO06_GPIO1_IO6		0x19	//Wake#
		>;
	};
};

* In imx8mqevk schematic, PCIe EP clk pins are connected to both clk_generator and imx8mq pcie clk pins(CLK_N, CLK_P). Do we really need both the conections? CLK from the clk_generator should be enough. Right?

Thanks

[joanxie]

In imx8mqevk schematic, PCIe EP clk pins are connected to both clk_generator and imx8mq pcie clk pins(CLK_N, CLK_P). Do we really need both the conections? CLK from the clk_generator should be enough. Right?

>what do you mean? what different between your schematic from imx8mqevk schematic?

[abhishekojhasfl]

Hi

In our schematic, 

1. PCIE clock generator chip (i.e. 9FGV0241) is feed clock to both the PCIe PHY and PCIe connecter/device. Clock generated from the clk generator connected to PCIEx_REF_PAD_CLK_P/N pins of imx8mq.

2. Not Connected - CLK2_P/N (ball T22 and U22) . The internal clock can be output from CLK2_P/N (ball T22 and U22) to provide clock to the connector/device.

In imx8mqevk schematic,

1. PCIE clock generator chip (i.e. 9FGV0241) is feed clock to both the PCIe PHY and PCIe connecter/device. Clock generated from the clk generator connected to PCIEx_REF_PAD_CLK_P/N pins of imx8mq.

2. Connected - CLK2_P/N (ball T22 and U22) to Connected device.

*Second point is the only difference.

 Thanks

[abhishekojhasfl]

Please find attached dts file. DTS file is based on imx8mq-evk.dts.

[joanxie]

why do you set pcieclk? what is for? if your design is based on imx8mq, pls refer to the evk dts as below

linux-imx/arch/arm64/boot/dts/freescale/imx8mq-evk.dts at lf-6.6.y · nxp-imx/linux-imx · GitHub

[abhishekojhasfl]

Initially, I used the imx8mq-evk.dts pcie node changes only but getting same result "Phy link never came up".

&pcie0 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_pcie0>;
	reset-gpio = <&gpio5 28 GPIO_ACTIVE_LOW>;
	clocks = <&clk IMX8MQ_CLK_PCIE1_ROOT>,
		 <&pcie0_refclk>,
		 <&clk IMX8MQ_CLK_PCIE1_PHY>,
		 <&clk IMX8MQ_CLK_PCIE1_AUX>;
	vph-supply = <&vgen5_reg>;
	status = "okay";
};

 With above configuration, PCIE_PHY clk is generated from system_pll2/10 and enabled. That means internal clk is getting used instead of CLK generator clk. This is my understanding.

devmem2 0x3038a380
/dev/mem opened.
Memory mapped at address 0xffffb850f000.
Read at address  0x3038A380 (0xffffb850f380): 0x11000000
bit(26-24)-001-SYSTEM_PLL2_DIV10 (CCM)
bit(28) - clk enable

I added "pcieclk" changes in device tree by referring https://elixir.bootlin.com/linux/v6.9.7/source/arch/arm64/boot/dts/freescale/imx8mq-tqma8mq-mba8mx.d...

imx8mq based platform.

But still same issue with "pcieclk" clk generator changes DTS changes.

GPR14 Register:
devmem2 0x30340038
/dev/mem opened.
Memory mapped at address 0xffff83957000.
Read at address  0x30340038 (0xffff83957038): 0x49408600
BIT(9) - PCIE_REF_USE_PAD
BIT(10) - PCIE_CLK_REQ_OVERRIDE_EN
BIT(11) - PCIE_CLK_REQ_OVERRIDE

[joanxie]

did you board design based on tqma8mq-mba8mx? it seems this board design isn't same as nxp imx8mq evk board, your dts settings is based on your HW design, for HW design, pls according to the HW design guide I sent to you before, if you couldn't sure of this, pls create a schematic review case to review your HW firstly, if your design is based on nxp evk board, pls refer to our dts, I also sent to you before, which is based on external clock, if you use the third party company board and SW, you need confirm from them, I don't know how about their pcie driver, current driver, if you read, you can find that this is for external clock as default, you also can measure the pcie clock to check if they are stable or not

https://github.com/nxp-imx/linux-imx/blob/lf-6.6.y/arch/arm64/boot/dts/freescale/imx8mq-evk.dts