Enabling All pcie in lS1043A

[Naveenkumar_Muthusamy]

Hi all,

we are working on software development using LS1043A with LA1043ARDB with pcie interface.

As per LS1043A Reference manual, this device supports three Pcie connections.

In the devictree also, we have seen three nodes of the pcie and all three has enabled attached Device tree as well. But once the board is booted we can only see two pcie ports enabled.

i have added dmesg logs and config.gz with this.

		pcie1: pcie@3400000 {
			compatible = "fsl,ls1043a-pcie";
			reg = <0x00 0x03400000 0x0 0x00100000>, /* controller registers */
			      <0x40 0x00000000 0x0 0x00002000>; /* configuration space */
			reg-names = "regs", "config";
			interrupts = <GIC_SPI 117 IRQ_TYPE_LEVEL_HIGH>,
				     <GIC_SPI 118 IRQ_TYPE_LEVEL_HIGH>;
			interrupt-names = "pme", "aer";
			#address-cells = <3>;
			#size-cells = <2>;
			device_type = "pci";
			iommu-map = <0 &smmu 0 1>;	/* update by bootloader */
			num-viewport = <6>;
			bus-range = <0x0 0xff>;
			ranges = <0x81000000 0x0 0x00000000 0x40 0x00010000 0x0 0x00010000   /* downstream I/O */
             			 0x82000000 0x0 0x40000000 0x40 0x40000000 0x0 0x40000000   /* 32-bit non-prefetchable */
      				0xc3000000 0x40 0x80000000 0x40 0x80000000 0x0 0x80000000>; /* 64-bit prefetchable memory */
			msi-parent = <&msi1>, <&msi2>, <&msi3>;
			#interrupt-cells = <1>;
			interrupt-map-mask = <0 0 0 7>;
			interrupt-map = <0000 0 0 1 &gic 0 110 0x4>,
					<0000 0 0 2 &gic 0 111 0x4>,
					<0000 0 0 3 &gic 0 112 0x4>,
					<0000 0 0 4 &gic 0 113 0x4>;
			fsl,pcie-scfg = <&scfg 0>;
			big-endian;
			status = "okay";
		};

		pcie2: pcie@3500000 {
			compatible = "fsl,ls1043a-pcie";
			reg = <0x00 0x03500000 0x0 0x00100000>, /* controller registers */
			      <0x48 0x00000000 0x0 0x00002000>; /* configuration space */
			reg-names = "regs", "config";
			interrupts = <GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH>,
				     <GIC_SPI 128 IRQ_TYPE_LEVEL_HIGH>;
			interrupt-names = "pme", "aer";
			#address-cells = <3>;
			#size-cells = <2>;
			device_type = "pci";
			iommu-map = <0 &smmu 0 1>;	/* update by bootloader */
			num-viewport = <6>;
			bus-range = <0x0 0xff>;
			/* ... existing properties ... */
   		 	ranges = <0x81000000 0x0 0x00000000 0x48 0x00010000 0x0 0x00010000   /* downstream I/O */
   		   		0x82000000 0x0 0x40000000 0x48 0x40000000 0x0 0x40000000   /* 32-bit non-prefetchable */
              			0xc3000000 0x48 0x80000000 0x48 0x80000000 0x0 0x80000000>; /* 64-bit prefetchable memory */
			msi-parent = <&msi1>, <&msi2>, <&msi3>;
			#interrupt-cells = <1>;
			interrupt-map-mask = <0 0 0 7>;
			interrupt-map = <0000 0 0 1 &gic 0 120  0x4>,
					<0000 0 0 2 &gic 0 121 0x4>,
					<0000 0 0 3 &gic 0 122 0x4>,
					<0000 0 0 4 &gic 0 123 0x4>;
			fsl,pcie-scfg = <&scfg 1>;
			big-endian;
			status = "okay";
		};

		pcie3: pcie@3600000 {
			compatible = "fsl,ls1043a-pcie";
			reg = <0x00 0x03600000 0x0 0x00100000>, /* controller registers */
			      <0x50 0x00000000 0x0 0x00002000>; /* configuration space */
			reg-names = "regs", "config";
			interrupts = <GIC_SPI 161 IRQ_TYPE_LEVEL_HIGH>,
				     <GIC_SPI 162 IRQ_TYPE_LEVEL_HIGH>;
			interrupt-names = "pme", "aer";
			#address-cells = <3>;
			#size-cells = <2>;
			device_type = "pci";
			iommu-map = <0 &smmu 0 1>;	/* update by bootloader */
			num-viewport = <6>;
			bus-range = <0x0 0xff>;
			ranges = <0x81000000 0x0 0x00000000 0x50 0x00010000 0x0 0x00010000   /* downstream I/O */
              			0x82000000 0x0 0x40000000 0x50 0x40000000 0x0 0x40000000   /* 32-bit non-prefetchable */
              			0xc3000000 0x50 0x80000000 0x50 0x80000000 0x0 0x80000000>; /* 64-bit prefetchable memory */
			msi-parent = <&msi1>, <&msi2>, <&msi3>;
			#interrupt-cells = <1>;
			interrupt-map-mask = <0 0 0 7>;
			interrupt-map = <0000 0 0 1 &gic 0 154 0x4>,
					<0000 0 0 2 &gic 0 155 0x4>,
					<0000 0 0 3 &gic 0 156 0x4>,
					<0000 0 0 4 &gic 0 157 0x4>;
			fsl,pcie-scfg = <&scfg 2>;
			big-endian;
			status = "okay";
		};

[Naveenkumar_Muthusamy]

Hi @yipingwang ,

any update on this issue?

[yipingwang]

Would you please also provide the whole u-boot console log?

[Naveenkumar_Muthusamy]

NOTICE:  Fixed DDR on board

NOTICE:  2 GB DDR4, 32-bit, CL=11, ECC off
NOTICE:  BL2: v2.6(release):lf-5.15.52-2.1.0-rc1-6-gaaad28622
NOTICE:  BL2: Built : 03:27:25, Aug 24 2022
NOTICE:  BL2: Booting BL31
NOTICE:  BL31: v2.6(release):lf-5.15.52-2.1.0-rc1-6-gaaad28622
NOTICE:  BL31: Built : 03:27:25, Aug 24 2022
NOTICE:  Welcome to ls1043ardb BL31 Phase


U-Boot 2022.04+fsl+ge9cda0d932 (Sep 09 2022 - 06:55:53 +0000)

SoC:  LS1043AE Rev1.1 (0x87920011)
Clock Configuration:
       CPU0(A53):1600 MHz  CPU1(A53):1600 MHz  CPU2(A53):1600 MHz  
       CPU3(A53):1600 MHz  
       Bus:      400  MHz  DDR:      1600 MT/s  FMAN:     500  MHz
Reset Configuration Word (RCW):
       00000000: 08100010 0a000000 00000000 00000000
       00000010: 14550002 80004012 e0025000 c1002000
       00000020: 00000000 00000000 00000000 00038800
       00000030: 00000000 00001101 00000096 00000001
Model: LS1043A RDB Board
Board: LS1043ARDB, boot from vBank 0
CPLD:  V3.0
PCBA:  V7.0
SERDES Reference Clocks:
SD1_CLK1 = 156.25MHZ, SD1_CLK2 = 100.00MHZ
DRAM:  1.9 GiB (DDR4, 32-bit, CL=11, ECC off)
Using SERDES1 Protocol: 5205 (0x1455)
PCIe1: pcie@3400000 disabled
PCIe2: pcie@3500000 Root Complex: no link
PCIe3: pcie@3600000 Root Complex: x1 gen2
Firmware 'Microcode version 0.0.1 for LS1021a r1.0' for 1021 V1.0
QE: uploading microcode 'Microcode for LS1021a r1.0' version 0.0.1
Core:  39 devices, 14 uclasses, devicetree: separate
Flash: 128 MiB
NAND:  512 MiB
MMC:   FSL_SDHC: 0
Loading Environment from Flash... OK
EEPROM: NXID v1
In:    serial
Out:   serial
Err:   serial
SEC0:  RNG instantiated
Net:   Fman1: Uploading microcode version 106.4.18
eth0: fm1-mac1 [PRIME], eth1: fm1-mac2, eth2: fm1-mac3, eth3: fm1-mac4, eth4: fm1-mac5, eth5: fm1-mac6, eth6: fm1-mac9
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /ls1043ardb_boot.scr

[yipingwang]

You could see the following in your u-boot log.

PCIe1: pcie@3400000 disabled
PCIe2: pcie@3500000 Root Complex: no link
PCIe3: pcie@3600000 Root Complex: x1 gen2

PCIe1 is disabled in your RCW configuration.

You are using SERDES1 protocol 0x1455, please refer to "Table 33-1. Supported SerDes options" in LS1043ARM, in this configuration only PCIE2 and PCIE3 are enabled. You could configure RCW[SRDS_PRTCL_S1] as 0x1555, then all these PCIEs can be enabled.

[Naveenkumar_Muthusamy]

Hi @yipingwang ,

yes, i have also noticied it, but currently we are using the referece image, we have not modified anything. 
can you let me know how to make this changes?

[yipingwang]

In your yocto build environment, please refer to the following procedure to modify RCW to rebuild atf and firmware image.

1. $ bitbake rcw -c cleansstate
2. $ bitbake rcw -c patch
3. Please go to rcw source code folder build_ls1043ardb/tmp/work/ls1043ardb-fsl-linux/rcw/git-r0/git/ please modify IFC NOR flash RCW file ls1043ardb/ RR_FQPP_1455/rcw_1600/ rcw_1600.rcw  according to your requirement.
4. $bitbake rcw

       $ bitbake qoriq-atf -c cleansstate

       $ bitbake qoriq-atf

        $ bitbake qoriq-composite-firmware -c cleansstate

        $ bitbake qoriq-composite-firmware

[Naveenkumar_Muthusamy]

hi,

Thanks for providing the steps for compiling RCW.

how to flash the newly generated rcw. we use SD card boot. so, we can just flash the .wic file and it works or we have to change anywhere else?