[lx2160ardb_rev2] PCIe card can't be detected

[tsungmin_wang]

Hi all,

I have a lx2160ardb_rev2 board, and I plugged a PCIe card to the board.

When I booted the board, I found that the board doesn't detect the PCIe card.

NOTICE:  BL2: v1.5(release):LSDK-20.04
NOTICE:  BL2: Built : 05:31:48, Apr  9 2020
NOTICE:  UDIMM 18ADF2G72AZ-3G2E1
NOTICE:  DDR4 UDIMM with 2-rank 64-bit bus (x8)

NOTICE:  32 GB DDR4, 64-bit, CL=22, ECC on, 256B, CS0+CS1
NOTICE:  BL2: Booting BL31
NOTICE:  BL31: v1.5(release):LSDK-20.04
NOTICE:  BL31: Built : 05:32:16, Apr  9 2020
NOTICE:  Welcome to LX2160 BL31 Phase


U-Boot 2019.10-g3cd9bc3993 (Apr 09 2020 - 05:30:34 +0800)

SoC:  LX2160ACE Rev2.0 (0x87360020)
Clock Configuration:
       CPU0(A72):2000 MHz  CPU1(A72):2000 MHz  CPU2(A72):2000 MHz
       CPU3(A72):2000 MHz  CPU4(A72):2000 MHz  CPU5(A72):2000 MHz
       CPU6(A72):2000 MHz  CPU7(A72):2000 MHz  CPU8(A72):2000 MHz
       CPU9(A72):2000 MHz  CPU10(A72):2000 MHz  CPU11(A72):2000 MHz
       CPU12(A72):2000 MHz  CPU13(A72):2000 MHz  CPU14(A72):2000 MHz
       CPU15(A72):2000 MHz
       Bus:      700  MHz  DDR:      2900 MT/s
Reset Configuration Word (RCW):
       00000000: 50777738 24500050 00000000 00000000
       00000010: 00000000 0c010000 00000000 00000000
       00000020: 02e001a0 00002580 00000000 00000096
       00000030: 00000000 00000000 00000000 00000000
       00000040: 00000000 00000000 00000000 00000000
       00000050: 00000000 00000000 00000000 00000000
       00000060: 00000000 00000000 00027000 00000000
       00000070: 08b30010 00150020
Model: NXP Layerscape LX2160ARDB Board
Board: LX2160ACE Rev2.0-RDB, Board version: B, boot from SD
FPGA: v8.0
SERDES1 Reference: Clock1 = 161.13MHz Clock2 = 161.13MHz
SERDES2 Reference: Clock1 = 100MHz Clock2 = 100MHz
SERDES3 Reference: Clock1 = 100MHz Clock2 = 100MHz
VID: Core voltage after adjustment is at 851 mV
DRAM:  31.9 GiB
DDR    31.9 GiB (DDR4, 64-bit, CL=22, ECC on)
       DDR Controller Interleaving Mode: 256B
       DDR Chip-Select Interleaving Mode: CS0+CS1
Using SERDES1 Protocol: 19 (0x13)
Using SERDES2 Protocol: 5 (0x5)
Using SERDES3 Protocol: 2 (0x2)
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
Loading Environment from MMC... *** Warning - bad CRC, using default environment

EEPROM: NXID v1
In:    serial_pl01x
Out:   serial_pl01x
Err:   serial_pl01x
Net:
CS4223: Using software initialization...
CS4223: edc/gain/equalization settings: host: CX/4dB/4dB, line: SR/4dB/4dB
FSL_MDIO1:0 is connected to DPMAC5@25g-aui.  Reconnecting to DPMAC6@25g-aui
PCIe0: pcie@3400000 disabled
PCIe1: pcie@3500000 disabled
PCIe2: pcie@3600000 Root Complex: no link
PCIe3: pcie@3700000 disabled
PCIe4: pcie@3800000 Root Complex: no link
PCIe5: pcie@3900000 disabled
DPMAC2@xlaui4, DPMAC3@xgmii, DPMAC4@xgmii, DPMAC5@25g-aui, DPMAC6@25g-aui, DPMAC17@rgmii-id, DPMAC18@rgmii-id

MMC read: dev # 0, block # 20480, count 4608 ... 4608 blocks read: OK

MMC read: dev # 0, block # 28672, count 2048 ... 2048 blocks read: OK
crc32+
fsl-mc: Booting Management Complex ... SUCCESS
fsl-mc: Management Complex booted (version: 10.20.4, boot status: 0x1)

But I used pci to scan device. The device existed actually. 

And I found that the link status of the PCIe device was strange. It just looked like a gen 1 card.

Link Speed: 2.5 GT/s

Link width: x1

=> pci 2
Scanning PCI devices on bus 2
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
02.00.00   0x1957     0x8d80     Bridge device           0x04

=> pci display.b 02.00.00
00000000: 57 19 80 8d 06 00 10 00 20 00 04 06 08 00 01 00
00000010: 00 00 00 40 00 00 00 44 00 01 01 00 01 01 00 00
00000020: 00 49 f0 48 01 10 01 00 00 00 00 00 00 00 00 00
00000030: 00 00 00 00 40 00 00 00 00 00 00 48 ff 01 00 00

=> pci display.b 02.00.00 70
00000070: 10 b0 42 00 21 80 00 00 10 29 00 00 43 f4 73 00
00000080: 08 00 11 10 00 00 00 00 c0 03 40 00 00 00 00 00
00000090: 00 00 00 00 3f 04 00 00 00 00 00 00 0e 00 00 00
000000a0: 03 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00

I measured the clock generator of PCIe on the board, and there was no clock on the output.

I switched SW2_3 to 1.

SW2_3 (SW_SLOTCLK)

0 : PCIE slots clocked only when card installed.

1 : PCIE slots clocked always.

And the clock is generated, but our EE said, it can only switch to gen 2.

Now, I can detect the PCIe card, but still, the link status is "Link Speed: 2.5 GT/s and Link width: x1"

PCIe0: pcie@3400000 disabled
PCIe1: pcie@3500000 disabled
PCIe2: pcie@3600000 Root Complex: no link
PCIe3: pcie@3700000 disabled
PCIe4: pcie@3800000 Root Complex: x8 gen3
PCIe5: pcie@3900000 disabled
DPMAC2@xlaui4, DPMAC3@xgmii, DPMAC4@xgmii, DPMAC5@25g-aui, DPMAC6@25g-aui, DPMAC17@rgmii-id, DPMAC18@rgmii-id

MMC read: dev # 0, block # 20480, count 4608 ... 4608 blocks read: OK

MMC read: dev # 0, block # 28672, count 2048 ... 2048 blocks read: OK
crc32+
fsl-mc: Booting Management Complex ... SUCCESS
fsl-mc: Management Complex booted (version: 10.20.4, boot status: 0x1)
Hit any key to stop autoboot:  0
=> pci display.b 02.00.00 70
00000070: 10 b0 42 00 21 80 00 00 10 29 00 00 43 f4 73 00
00000080: 08 00 11 10 00 00 00 00 c0 03 40 00 00 00 00 00
00000090: 00 00 00 00 3f 04 00 00 00 00 00 00 0e 00 00 00
000000a0: 03 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00

 When the PCIe card is detected, I try to boot into userspace, but SError is occured.

=> run bootcmd_mmc0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Scanning mmc 0:2...
Found U-Boot script /lx2160ardb_boot.scr
980 bytes read in 19 ms (49.8 KiB/s)
## Executing script at 80000000
36745728 bytes read in 1759 ms (19.9 MiB/s)
34033 bytes read in 19 ms (1.7 MiB/s)
## Flattened Device Tree blob at 90000000
   Booting using the fdt blob at 0x90000000
   Loading Device Tree to 000000009fff4000, end 000000009ffff4f0 ... OK
fsl-mc: DPL not deployed, DPAA2 ethernet not work

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.4.3 (root@workspace) (gcc version 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)) #1 SMP PREEMPT Mon Nov 9 08:46:39 CST 2020
[    0.000000] Machine model: NXP Layerscape LX2160ARDB
[    0.000000] earlycon: pl11 at MMIO32 0x00000000021c0000 (options '')
[    0.000000] printk: bootconsole [pl11] enabled
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 320 MiB at 0x00000000e7c00000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000277fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x277c1dc800-0x277c1ddfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000277fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000fbdfffff]
[    0.000000]   node   0: [mem 0x0000002080000000-0x000000277fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000277fffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] percpu: Embedded 24 pages/cpu s59928 r8192 d30184 u98304
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: EL2 vector hardening
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] Speculative Store Bypass Disable mitigation not required
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 7724544
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=ttyAMA0,115200 earlycon=pl011,mmio32,0x21c0000 root=PARTUUID=071d27c6-04 rw rootwait pci=pcie_bus_perf
[    0.000000] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.000000] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0xe3c00000-0xe7c00000] (64MB)
[    0.000000] Memory: 30353236K/31389696K available (18620K kernel code, 2476K rwdata, 8848K rodata, 5888K init, 993K bss, 708780K reserved, 327680K cma-reserved)
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000]  Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: no VLPI support, no direct LPI support
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006200000
[    0.000000] ITS [mem 0x06020000-0x0603ffff]
[    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @275a880000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: Using preallocated redistributor tables
[    0.000000] GICv3: using LPI property table @0x000000277fe00000
[    0.000000] GICv3: CPU0: using reserved LPI pending table @0x000000277fe10000
[    0.000000] random: get_random_bytes called from start_kernel+0x2b8/0x458 with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c409fb33, max_idle_ns: 440795203156 ns
[    0.000001] sched_clock: 56 bits at 25MHz, resolution 39ns, wraps every 4398046511103ns
[    0.008411] Console: colour dummy device 80x25
[    0.012942] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.023409] pid_max: default: 32768 minimum: 301
[    0.028151] LSM: Security Framework initializing
[    0.032944] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.040686] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.072817] ASID allocator initialised with 32768 entries
[    0.086283] rcu: Hierarchical SRCU implementation.
[    0.099158] Platform MSI: gic-its@6020000 domain created
[    0.104568] PCI/MSI: /interrupt-controller@6000000/gic-its@6020000 domain created
[    0.112158] fsl-mc MSI: /interrupt-controller@6000000/gic-its@6020000 domain created
[    0.120398] EFI services will not be available.
[    0.132983] smp: Bringing up secondary CPUs ...
[    0.169700] Detected PIPT I-cache on CPU1
[    0.169716] GICv3: CPU1: found redistributor 1 region 0:0x0000000006220000
[    0.169728] GICv3: CPU1: using reserved LPI pending table @0x000000277fe20000
[    0.169846] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.201716] Detected PIPT I-cache on CPU2
[    0.201731] GICv3: CPU2: found redistributor 100 region 0:0x0000000006240000
[    0.201744] GICv3: CPU2: using reserved LPI pending table @0x000000277fe30000
[    0.201860] CPU2: Booted secondary processor 0x0000000100 [0x410fd083]
[    0.233737] Detected PIPT I-cache on CPU3
[    0.233748] GICv3: CPU3: found redistributor 101 region 0:0x0000000006260000
[    0.233758] GICv3: CPU3: using reserved LPI pending table @0x000000277fe40000
[    0.233871] CPU3: Booted secondary processor 0x0000000101 [0x410fd083]
[    0.265769] Detected PIPT I-cache on CPU4
[    0.265783] GICv3: CPU4: found redistributor 200 region 0:0x0000000006280000
[    0.265795] GICv3: CPU4: using reserved LPI pending table @0x000000277fe50000
[    0.265911] CPU4: Booted secondary processor 0x0000000200 [0x410fd083]
[    0.297791] Detected PIPT I-cache on CPU5
[    0.297803] GICv3: CPU5: found redistributor 201 region 0:0x00000000062a0000
[    0.297813] GICv3: CPU5: using reserved LPI pending table @0x000000277fe60000
[    0.297926] CPU5: Booted secondary processor 0x0000000201 [0x410fd083]
[    0.329823] Detected PIPT I-cache on CPU6
[    0.329839] GICv3: CPU6: found redistributor 300 region 0:0x00000000062c0000
[    0.329852] GICv3: CPU6: using reserved LPI pending table @0x000000277fe70000
[    0.329968] CPU6: Booted secondary processor 0x0000000300 [0x410fd083]
[    0.361847] Detected PIPT I-cache on CPU7
[    0.361861] GICv3: CPU7: found redistributor 301 region 0:0x00000000062e0000
[    0.361871] GICv3: CPU7: using reserved LPI pending table @0x000000277fe80000
[    0.361982] CPU7: Booted secondary processor 0x0000000301 [0x410fd083]
[    0.393879] Detected PIPT I-cache on CPU8
[    0.393897] GICv3: CPU8: found redistributor 400 region 0:0x0000000006300000
[    0.393909] GICv3: CPU8: using reserved LPI pending table @0x000000277fe90000
[    0.394026] CPU8: Booted secondary processor 0x0000000400 [0x410fd083]
[    0.425902] Detected PIPT I-cache on CPU9
[    0.425918] GICv3: CPU9: found redistributor 401 region 0:0x0000000006320000
[    0.425928] GICv3: CPU9: using reserved LPI pending table @0x000000277fea0000
[    0.426041] CPU9: Booted secondary processor 0x0000000401 [0x410fd083]
[    0.457944] Detected PIPT I-cache on CPU10
[    0.457963] GICv3: CPU10: found redistributor 500 region 0:0x0000000006340000
[    0.457976] GICv3: CPU10: using reserved LPI pending table @0x000000277feb0000
[    0.458093] CPU10: Booted secondary processor 0x0000000500 [0x410fd083]
[    0.489959] Detected PIPT I-cache on CPU11
[    0.489976] GICv3: CPU11: found redistributor 501 region 0:0x0000000006360000
[    0.489986] GICv3: CPU11: using reserved LPI pending table @0x000000277fec0000
[    0.490097] CPU11: Booted secondary processor 0x0000000501 [0x410fd083]
[    0.521991] Detected PIPT I-cache on CPU12
[    0.522012] GICv3: CPU12: found redistributor 600 region 0:0x0000000006380000
[    0.522025] GICv3: CPU12: using reserved LPI pending table @0x000000277fed0000
[    0.522141] CPU12: Booted secondary processor 0x0000000600 [0x410fd083]
[    0.554014] Detected PIPT I-cache on CPU13
[    0.554032] GICv3: CPU13: found redistributor 601 region 0:0x00000000063a0000
[    0.554043] GICv3: CPU13: using reserved LPI pending table @0x000000277fee0000
[    0.554157] CPU13: Booted secondary processor 0x0000000601 [0x410fd083]
[    0.586047] Detected PIPT I-cache on CPU14
[    0.586070] GICv3: CPU14: found redistributor 700 region 0:0x00000000063c0000
[    0.586083] GICv3: CPU14: using reserved LPI pending table @0x000000277fef0000
[    0.586201] CPU14: Booted secondary processor 0x0000000700 [0x410fd083]
[    0.618069] Detected PIPT I-cache on CPU15
[    0.618088] GICv3: CPU15: found redistributor 701 region 0:0x00000000063e0000
[    0.618099] GICv3: CPU15: using reserved LPI pending table @0x000000277ff00000
[    0.618210] CPU15: Booted secondary processor 0x0000000701 [0x410fd083]
[    0.618249] smp: Brought up 1 node, 16 CPUs
[    0.998995] SMP: Total of 16 processors activated.
[    1.003867] CPU features: detected: 32-bit EL0 Support
[    1.009068] CPU features: detected: CRC32 instructions
[    1.025867] CPU: All CPU(s) started at EL2
[    1.030033] alternatives: patching kernel code
[    1.035240] devtmpfs: initialized
[    1.040922] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.050780] futex hash table entries: 4096 (order: 6, 262144 bytes, linear)
[    1.060367] xor: measuring software checksum speed
[    1.105238]    8regs     :  8458.000 MB/sec
[    1.149479]    32regs    :  8633.000 MB/sec
[    1.193721]    arm64_neon:  8465.000 MB/sec
[    1.197944] xor: using function: 32regs (8633.000 MB/sec)
[    1.203405] pinctrl core: initialized pinctrl subsystem
[    1.209111] DMI not present or invalid.
[    1.213143] NET: Registered protocol family 16
[    1.218232] DMA: preallocated 256 KiB pool for atomic allocations
[    1.224396] audit: initializing netlink subsys (disabled)
[    1.229925] audit: type=2000 audit(1.028:1): state=initialized audit_enabled=0 res=1
[    1.237766] cpuidle: using governor menu
[    1.241880] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    1.249297] Serial: AMBA PL011 UART driver
[    1.253475] imx mu driver is registered.
[    1.257444] imx rpmsg driver is registered.
[    1.262523] Machine: NXP Layerscape LX2160ARDB
[    1.267012] SoC family: QorIQ LX2160A
[    1.270706] SoC ID: svr:0x87360020, Revision: 2.0
[    1.276578] 21c0000.serial: ttyAMA0 at MMIO 0x21c0000 (irq = 20, base_baud = 0) is a SBSA
[    1.284851] printk: console [ttyAMA0] enabled
[    1.284851] printk: console [ttyAMA0] enabled
[    1.293593] printk: bootconsole [pl11] disabled
[    1.293593] printk: bootconsole [pl11] disabled
[    1.302934] sbsa-uart 21d0000.serial: aliased and non-aliased serial devices found in device tree. Serial port enumeration may be unpredictable.
[    1.315879] 21d0000.serial: ttyAMA1 at MMIO 0x21d0000 (irq = 21, base_baud = 0) is a SBSA
[    1.333961] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    1.340659] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    1.347349] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    1.354040] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    1.362023] cryptd: max_cpu_qlen set to 1000
[    1.434324] raid6: neonx8   gen()  4865 MB/s
[    1.506359] raid6: neonx8   xor()  4163 MB/s
[    1.578403] raid6: neonx4   gen()  4007 MB/s
[    1.650425] raid6: neonx4   xor()  3865 MB/s
[    1.722462] raid6: neonx2   gen()  3202 MB/s
[    1.794491] raid6: neonx2   xor()  3646 MB/s
[    1.866531] raid6: neonx1   gen()  2454 MB/s
[    1.938556] raid6: neonx1   xor()  2744 MB/s
[    2.010600] raid6: int64x8  gen()  2171 MB/s
[    2.082639] raid6: int64x8  xor()  1791 MB/s
[    2.154936] raid6: int64x4  gen()  2529 MB/s
[    2.227238] raid6: int64x4  xor()  1809 MB/s
[    2.299570] raid6: int64x2  gen()  1683 MB/s
[    2.371876] raid6: int64x2  xor()  1635 MB/s
[    2.444184] raid6: int64x1  gen()  1229 MB/s
[    2.516494] raid6: int64x1  xor()  1105 MB/s
[    2.520754] raid6: using algorithm neonx8 gen() 4865 MB/s
[    2.526142] raid6: .... xor() 4163 MB/s, rmw enabled
[    2.531095] raid6: using neon recovery algorithm
[    2.535919] ACPI: Interpreter disabled.
[    2.540305] iommu: Default domain type: Passthrough
[    2.545325] vgaarb: loaded
[    2.548163] SCSI subsystem initialized
[    2.552058] usbcore: registered new interface driver usbfs
[    2.557550] usbcore: registered new interface driver hub
[    2.562896] usbcore: registered new device driver usb
[    2.568117] imx-i2c 2000000.i2c: can't get pinctrl, bus recovery not supported
[    2.575411] i2c i2c-0: IMX I2C adapter registered
[    2.580179] imx-i2c 2040000.i2c: can't get pinctrl, bus recovery not supported
[    2.587455] i2c i2c-1: IMX I2C adapter registered
[    2.592332] mc: Linux media interface: v0.10
[    2.596604] videodev: Linux video capture interface: v2.00
[    2.602106] pps_core: LinuxPPS API ver. 1 registered
[    2.607061] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    2.616186] PTP clock support registered
[    2.620134] EDAC MC: Ver: 3.0.0
[    2.623639] No BMan portals available!
[    2.627531] QMan: Allocated lookup table at (____ptrval____), entry count 65537
[    2.634902] No QMan portals available!
[    2.638735] No USDPAA memory, no 'fsl,usdpaa-mem' in device-tree
[    2.644907] FPGA manager framework
[    2.648332] Advanced Linux Sound Architecture Driver Initialized.
[    2.654835] clocksource: Switched to clocksource arch_sys_counter
[    2.661007] VFS: Disk quotas dquot_6.6.0
[    2.664954] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.671894] pnp: PnP ACPI: disabled
[    2.677973] thermal_sys: Registered thermal governor 'step_wise'
[    2.677975] thermal_sys: Registered thermal governor 'power_allocator'
[    2.684391] NET: Registered protocol family 2
[    2.695459] tcp_listen_portaddr_hash hash table entries: 16384 (order: 6, 262144 bytes, linear)
[    2.704315] TCP established hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    2.713191] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    2.720978] TCP: Hash tables configured (established 262144 bind 65536)
[    2.727660] UDP hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    2.734835] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    2.742567] NET: Registered protocol family 1
[    2.747111] RPC: Registered named UNIX socket transport module.
[    2.753025] RPC: Registered udp transport module.
[    2.757719] RPC: Registered tcp transport module.
[    2.762412] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    2.768845] PCI: CLS 0 bytes, default 64
[    2.773498] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    2.781768] kvm [1]: IPA Size Limit: 44bits
[    2.786297] kvm [1]: vgic-v2@c0e0000
[    2.789876] kvm [1]: GIC system register CPU interface enabled
[    2.795863] kvm [1]: vgic interrupt IRQ1
[    2.799946] kvm [1]: Hyp mode initialized successfully
[    2.808581] Initialise system trusted keyrings
[    2.813081] workingset: timestamp_bits=44 max_order=23 bucket_order=0
[    2.819959] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    2.825946] NFS: Registering the id_resolver key type
[    2.831000] Key type id_resolver registered
[    2.835174] Key type id_legacy registered
[    2.839177] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    2.845877] jffs2: version 2.2. (NAND) c 2001-2006 Red Hat, Inc.
[    2.852059] fuse: init (API version 7.31)
[    2.856150] 9p: Installing v9fs 9p2000 file system support
[    2.868634] Key type asymmetric registered
[    2.872724] Asymmetric key parser 'x509' registered
[    2.877603] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    2.885055] io scheduler mq-deadline registered
[    2.889579] io scheduler kyber registered
[    2.896741] layerscape-pcie 3600000.pcie: host bridge /soc/pcie@3600000 ranges:
[    2.904052] layerscape-pcie 3600000.pcie:   MEM 0x9040000000..0x907fffffff -> 0x40000000
[    2.912211] layerscape-pcie 3600000.pcie: PCI host bridge to bus 0000:00
[    2.918905] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.924382] pci_bus 0000:00: root bus resource [mem 0x9040000000-0x907fffffff] (bus address [0x40000000-0x7fffffff])
[    2.934905] pci 0000:00:00.0: [1957:8d80] type 01 class 0x060400
[    2.940918] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    2.947180] pci 0000:00:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
[    2.953441] pci 0000:00:00.0: reg 0x38: [mem 0x9048000000-0x9048ffffff pref]
[    2.960525] pci 0000:00:00.0: supports D1 D2
[    2.964786] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    2.971881] pci 0000:00:00.0: BAR 1: assigned [mem 0x9040000000-0x9043ffffff]
[    2.979010] pci 0000:00:00.0: BAR 0: assigned [mem 0x9044000000-0x9044ffffff]
[    2.986137] pci 0000:00:00.0: BAR 6: assigned [mem 0x9045000000-0x9045ffffff pref]
[    2.993697] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    2.998926] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
[    3.007430] layerscape-pcie 3800000.pcie: host bridge /soc/pcie@3800000 ranges:
[    3.014737] layerscape-pcie 3800000.pcie:   MEM 0xa040000000..0xa07fffffff -> 0x40000000
[    3.022897] layerscape-pcie 3800000.pcie: PCI host bridge to bus 0001:00
[    3.029590] pci_bus 0001:00: root bus resource [bus 00-ff]
[    3.035067] pci_bus 0001:00: root bus resource [mem 0xa040000000-0xa07fffffff] (bus address [0x40000000-0x7fffffff])
[    3.045588] pci 0001:00:00.0: [1957:8d80] type 01 class 0x060400
[    3.051598] pci 0001:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.057860] pci 0001:00:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
[    3.064121] pci 0001:00:00.0: reg 0x38: [mem 0xa048000000-0xa048ffffff pref]
[    3.071205] pci 0001:00:00.0: supports D1 D2
[    3.075466] pci 0001:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.082170] pci 0001:01:00.0: [10b5:8724] type 01 class 0x060400
[    3.088488] pci 0001:01:00.0: reg 0x10: [mem 0xa049000000-0xa04903ffff]
[    3.096352] pci 0001:01:00.0: PME# supported from D0 D3hot D3cold
[    3.104185] SError Interrupt on CPU0, code 0xbf000002 -- SError
[    3.104186] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.3 #1
[    3.104187] Hardware name: NXP Layerscape LX2160ARDB (DT)
[    3.104188] pstate: 60000085 (nZCv daIf -PAN -UAO)
[    3.104189] pc : dw_pcie_read+0x80/0xa0
[    3.104189] lr : dw_pcie_access_other_conf.isra.16+0xd0/0xe8
[    3.104190] sp : ffff80001009b660
[    3.104190] x29: ffff80001009b660 x28: ffff80001009b954
[    3.104192] x27: ffff0026d9121800 x26: 0000000000000000
[    3.104194] x25: ffff80001003d800 x24: 0000000000000000
[    3.104195] x23: ffff80001009b754 x22: 0000000000000004
[    3.104197] x21: ffff0026d9111480 x20: 0000000000000000
[    3.104198] x19: ffff0026d91114a8 x18: ffffffffffffffff
[    3.104200] x17: 000000006d32ea7b x16: 00000000f9c7e68a
[    3.104201] x15: ffffd65d7d729908 x14: ffff0026d912491c
[    3.104203] x13: ffff0026d91241a4 x12: 0000000000000038
[    3.104204] x11: 0000000000000004 x10: 0101010101010101
[    3.104206] x9 : 00000000872410b5 x8 : 0000000000000908
[    3.104207] x7 : 0000000000000908 x6 : ffff800010f00000
[    3.104208] x5 : 0000000000000000 x4 : ffff0026d9111480
[    3.104210] x3 : 0000000000000003 x2 : ffff80001009b754
[    3.104211] x1 : 0000000000000000 x0 : ffff80001003d800
[    3.104213] Kernel panic - not syncing: Asynchronous SError Interrupt
[    3.104214] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.3 #1
[    3.104214] Hardware name: NXP Layerscape LX2160ARDB (DT)
[    3.104215] Call trace:
[    3.104216]  dump_backtrace+0x0/0x150
[    3.104216]  show_stack+0x14/0x20
[    3.104217]  dump_stack+0xbc/0x100
[    3.104217]  panic+0x16c/0x37c
[    3.104218]  __stack_chk_fail+0x0/0x18
[    3.104219]  arm64_serror_panic+0x74/0x88
[    3.104219]  do_serror+0x70/0x138
[    3.104220]  el1_error+0x84/0xf8
[    3.104220]  dw_pcie_read+0x80/0xa0
[    3.104221]  dw_pcie_rd_conf+0x120/0x128
[    3.104222]  pci_bus_read_config_dword+0x84/0xe0
[    3.104222]  pci_bus_generic_read_dev_vendor_id+0x2c/0x1a0
[    3.104223]  pci_bus_read_dev_vendor_id+0x48/0x68
[    3.104223]  pci_scan_single_device+0x78/0xf8
[    3.104224]  pci_scan_slot+0x34/0x128
[    3.104225]  pci_scan_child_bus_extend+0x50/0x290
[    3.104225]  pci_scan_child_bus+0x10/0x18
[    3.104226]  pci_scan_bridge_extend+0x228/0x560
[    3.104227]  pci_scan_child_bus_extend+0x12c/0x290
[    3.104227]  pci_scan_child_bus+0x10/0x18
[    3.104228]  pci_scan_bridge_extend+0x228/0x560
[    3.104228]  pci_scan_child_bus_extend+0x12c/0x290
[    3.104229]  pci_scan_root_bus_bridge+0x58/0xe8
[    3.104230]  dw_pcie_host_init+0x2fc/0x5e8
[    3.104230]  ls_pcie_probe+0x100/0x134
[    3.104231]  platform_drv_probe+0x50/0xa0
[    3.104231]  really_probe+0x108/0x348
[    3.104232]  driver_probe_device+0x58/0x100
[    3.104232]  device_driver_attach+0x6c/0x90
[    3.104233]  __driver_attach+0x84/0xc8
[    3.104233]  bus_for_each_dev+0x74/0xc8
[    3.104234]  driver_attach+0x20/0x28
[    3.104235]  bus_add_driver+0x148/0x1f0
[    3.104235]  driver_register+0x60/0x110
[    3.104236]  __platform_driver_probe+0x74/0x130
[    3.104236]  ls_pcie_driver_init+0x20/0x28
[    3.104237]  do_one_initcall+0x5c/0x1b0
[    3.104237]  kernel_init_freeable+0x1a4/0x24c
[    3.104238]  kernel_init+0x10/0x108
[    3.104238]  ret_from_fork+0x10/0x18
[    3.104260] SMP: stopping secondary CPUs
[    3.104261] Kernel Offset: 0x565d6b600000 from 0xffff800010000000
[    3.104262] PHYS_OFFSET: 0xffff938f40000000
[    3.104262] CPU features: 0x0002,21806008
[    3.104263] Memory Limit: none

I can't boot into userspace when the PCIe card is detected. (Both link speed and link width are incorrect)

Is there any advise? Thanks.

[mks3134]

Hi Sir,

Is this resolved as I am still seeing this ?

Regards,

MVS

[mks3134]

Hi Sir,

Is this issue resolved?

Regards,

MVS

[ufedor]

You wrote:

> Now, I can detect the PCIe card, but still, the link status is "Link Speed: 2.5 GT/s and Link width: x1"

Excuse me, how this statement corresponds with the following line from the U-Boot log:

PCIe4: pcie@3800000 Root Complex: x8 gen3

?

Which exactly PCIe card is in question?

Is it capable to operate properly in a Windows desktop?

What is the LX2160ARDB SW2[1] setting?

[tsungmin_wang]

Hi ufedor,

I use the following command to read register from PCIe device.

=> pci display.b 02.00.00 70
00000070: 10 b0 42 00 21 80 00 00 10 29 00 00 43 f4 73 00
00000080: 08 00 11 10 00 00 00 00 c0 03 40 00 00 00 00 00
00000090: 00 00 00 00 3f 04 00 00 00 00 00 00 0e 00 00 00
000000a0: 03 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00

According to (10h) PCI Express Capability Structure under PCI Configuration Space, here is the reference.

I can get the Link status of the PCIe device.

00000080: 08 00 11 10

Link status [15:0] = 0001 0000 0001 0001 (10 11)

Link status [3:0] = 0001 ( = 2.5 GT/s )

Link status [9:4] = 00 0001 ( = x1 )

This PCIe device is network card, and it can run normally on other devices.

But there are many problems when I want to use it on lx2160ardb_rev2.

First, when I use default DIP setting from user guide (boot from SD)

Boot source	SW1[1:8]	SW2[1:8]	SW3[1:8]	SW4[1:8]
SD Card (SDHC1)	1000 1000	0000 0110	1111 1100	1011 1000

And the PCIe card can't be detected under u-boot

PCIe0: pcie@3400000 disabled
PCIe1: pcie@3500000 disabled
PCIe2: pcie@3600000 Root Complex: no link
PCIe3: pcie@3700000 disabled
PCIe4: pcie@3800000 Root Complex: no link
PCIe5: pcie@3900000 disabled

Although the PCIe card is not detected, I can use pci command to scan the device.

=> pci 2
Scanning PCI devices on bus 2
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
02.00.00   0x1957     0x8d80     Bridge device           0x04

=> pci display.b 02.00.00 70
00000070: 10 b0 42 00 21 80 00 00 10 29 00 00 43 f4 73 00
00000080: 08 00 11 10 00 00 00 00 c0 03 40 00 00 00 00 00

When I switch SW2[3] to 1, the PCIe card can be detected.

PCIe0: pcie@3400000 disabled
PCIe1: pcie@3500000 disabled
PCIe2: pcie@3600000 Root Complex: no link
PCIe3: pcie@3700000 disabled
PCIe4: pcie@3800000 Root Complex: x8 gen3
PCIe5: pcie@3900000 disabled

But the Link status is still same (Link Speed: 2.5 GT/s and Link width: x1).

And if I boot into userspace, it will occur SError.

=> pci display.b 02.00.00 70
00000070: 10 b0 42 00 21 80 00 00 10 29 00 00 43 f4 73 00
00000080: 08 00 11 10 00 00 00 00 c0 03 40 00 00 00 00 00

It looks like the PCIe card can't switch to Gen 3 on lx2160ardb_rev2 to work properly.

Here is the value of the register when this PCIe card is installed on another device.

00000060: 00 00 00 00 00 00 00 00 10 a4 52 00 04 80 00 00
00000070: 10 08 09 00 83 60 41 09 00 00 41 00 00 00 00 00

(PCI Express Capability Structure is start at 0x68 on another device, at 0x70 on lx2160ardb_rev2)

00000070: 10 08 09 00 83 60 41 09 00 00 41 00 00 00 00 00

Link status [15:0] = 0110 0000 1000 0011 (60 83)

Link status [3:0] = 0011 ( = 8.0 GT/s )

Link status [9:4] = 00 1000 ( = x8 )

[ufedor]

=> pci 2
Scanning PCI devices on bus 2
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
02.00.00   0x1957     0x8d80     Bridge device           0x04

Actually this means that no external devices are present on the bus 2.

The VendorId 0x1957 belongs to the LX2160A PCIe controller.

When the PCIe card is installed please provide logs for:

pci 1

pci 3

pci 4

pci 5

[tsungmin_wang]

Hi,

Sorry for my mistake, the link speed is checked, and it is correct.

Now, the question is I can't boot into userspace when the PCIe card is installed.

The error is occur when running this function.

[    3.121840] pc : dw_pcie_read+0x80/0xa0

Here is the log of the PCIe card under u-boot when the card is installed on lx2160ardb_rev2.

(U-Boot)
=> pci 6
Scanning PCI devices on bus 6
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
06.00.00   0x10b5     0x8724     Bridge device           0x04

Loading Kernel Log:

=> run bootcmd
Device: FSL_SDHC
Manufacturer ID: 1d
OEM: 4144
Name: SD
Bus Speed: 50000000
Mode: SD High Speed (50MHz)
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 57.8 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes

MMC read: dev # 0, block # 26624, count 2048 ... 2048 blocks read: OK
starting USB...
Bus usb3@3100000: Register 200017f NbrPorts 2
Starting the controller
USB XHCI 1.00
Bus usb3@3110000: Register 200017f NbrPorts 2
Starting the controller
USB XHCI 1.00
scanning bus usb3@3100000 for devices... 1 USB Device(s) found
scanning bus usb3@3110000 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found

Device 0: unknown device
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Scanning mmc 0:2...
Found U-Boot script /lx2160ardb_boot.scr
980 bytes read in 18 ms (52.7 KiB/s)
## Executing script at 80000000
36745728 bytes read in 1759 ms (19.9 MiB/s)
34033 bytes read in 19 ms (1.7 MiB/s)
## Flattened Device Tree blob at 90000000
   Booting using the fdt blob at 0x90000000
   Loading Device Tree to 000000009fff4000, end 000000009ffff4f0 ... OK
fsl-mc: Deploying data path layout ... SUCCESS

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.4.3 (root@workspace) (gcc version 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)) #1 SMP PREEMPT Mon Nov 9 08:46:39 CST 2020
[    0.000000] Machine model: NXP Layerscape LX2160ARDB
[    0.000000] earlycon: pl11 at MMIO32 0x00000000021c0000 (options '')
[    0.000000] printk: bootconsole [pl11] enabled
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 320 MiB at 0x00000000e7c00000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000277fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x277c1dc800-0x277c1ddfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000277fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000fbdfffff]
[    0.000000]   node   0: [mem 0x0000002080000000-0x000000277fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000277fffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] percpu: Embedded 24 pages/cpu s59928 r8192 d30184 u98304
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: EL2 vector hardening
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] Speculative Store Bypass Disable mitigation not required
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 7724544
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=ttyAMA0,115200 earlycon=pl011,mmio32,0x21c0000 root=PARTUUID=071d27c6-04 rw rootwait pci=pcie_bus_perf
[    0.000000] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.000000] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0xe3c00000-0xe7c00000] (64MB)
[    0.000000] Memory: 30353236K/31389696K available (18620K kernel code, 2476K rwdata, 8848K rodata, 5888K init, 993K bss, 708780K reserved, 327680K cma-reserved)
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000]  Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: no VLPI support, no direct LPI support
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006200000
[    0.000000] ITS [mem 0x06020000-0x0603ffff]
[    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @275a880000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: Using preallocated redistributor tables
[    0.000000] GICv3: using LPI property table @0x000000277fe00000
[    0.000000] GICv3: CPU0: using reserved LPI pending table @0x000000277fe10000
[    0.000000] random: get_random_bytes called from start_kernel+0x2b8/0x458 with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c409fb33, max_idle_ns: 440795203156 ns
[    0.000001] sched_clock: 56 bits at 25MHz, resolution 39ns, wraps every 4398046511103ns
[    0.008405] Console: colour dummy device 80x25
[    0.012934] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.023400] pid_max: default: 32768 minimum: 301
[    0.028141] LSM: Security Framework initializing
[    0.032933] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.040673] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.072803] ASID allocator initialised with 32768 entries
[    0.086268] rcu: Hierarchical SRCU implementation.
[    0.099144] Platform MSI: gic-its@6020000 domain created
[    0.104552] PCI/MSI: /interrupt-controller@6000000/gic-its@6020000 domain created
[    0.112141] fsl-mc MSI: /interrupt-controller@6000000/gic-its@6020000 domain created
[    0.120380] EFI services will not be available.
[    0.132965] smp: Bringing up secondary CPUs ...
[    0.169680] Detected PIPT I-cache on CPU1
[    0.169697] GICv3: CPU1: found redistributor 1 region 0:0x0000000006220000
[    0.169709] GICv3: CPU1: using reserved LPI pending table @0x000000277fe20000
[    0.169823] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.201695] Detected PIPT I-cache on CPU2
[    0.201710] GICv3: CPU2: found redistributor 100 region 0:0x0000000006240000
[    0.201722] GICv3: CPU2: using reserved LPI pending table @0x000000277fe30000
[    0.201839] CPU2: Booted secondary processor 0x0000000100 [0x410fd083]
[    0.233715] Detected PIPT I-cache on CPU3
[    0.233726] GICv3: CPU3: found redistributor 101 region 0:0x0000000006260000
[    0.233736] GICv3: CPU3: using reserved LPI pending table @0x000000277fe40000
[    0.233847] CPU3: Booted secondary processor 0x0000000101 [0x410fd083]
[    0.265746] Detected PIPT I-cache on CPU4
[    0.265760] GICv3: CPU4: found redistributor 200 region 0:0x0000000006280000
[    0.265772] GICv3: CPU4: using reserved LPI pending table @0x000000277fe50000
[    0.265886] CPU4: Booted secondary processor 0x0000000200 [0x410fd083]
[    0.297768] Detected PIPT I-cache on CPU5
[    0.297780] GICv3: CPU5: found redistributor 201 region 0:0x00000000062a0000
[    0.297791] GICv3: CPU5: using reserved LPI pending table @0x000000277fe60000
[    0.297901] CPU5: Booted secondary processor 0x0000000201 [0x410fd083]
[    0.329799] Detected PIPT I-cache on CPU6
[    0.329815] GICv3: CPU6: found redistributor 300 region 0:0x00000000062c0000
[    0.329827] GICv3: CPU6: using reserved LPI pending table @0x000000277fe70000
[    0.329940] CPU6: Booted secondary processor 0x0000000300 [0x410fd083]
[    0.361822] Detected PIPT I-cache on CPU7
[    0.361835] GICv3: CPU7: found redistributor 301 region 0:0x00000000062e0000
[    0.361846] GICv3: CPU7: using reserved LPI pending table @0x000000277fe80000
[    0.361956] CPU7: Booted secondary processor 0x0000000301 [0x410fd083]
[    0.393854] Detected PIPT I-cache on CPU8
[    0.393872] GICv3: CPU8: found redistributor 400 region 0:0x0000000006300000
[    0.393884] GICv3: CPU8: using reserved LPI pending table @0x000000277fe90000
[    0.394000] CPU8: Booted secondary processor 0x0000000400 [0x410fd083]
[    0.425878] Detected PIPT I-cache on CPU9
[    0.425893] GICv3: CPU9: found redistributor 401 region 0:0x0000000006320000
[    0.425904] GICv3: CPU9: using reserved LPI pending table @0x000000277fea0000
[    0.426016] CPU9: Booted secondary processor 0x0000000401 [0x410fd083]
[    0.457917] Detected PIPT I-cache on CPU10
[    0.457936] GICv3: CPU10: found redistributor 500 region 0:0x0000000006340000
[    0.457948] GICv3: CPU10: using reserved LPI pending table @0x000000277feb0000
[    0.458062] CPU10: Booted secondary processor 0x0000000500 [0x410fd083]
[    0.489932] Detected PIPT I-cache on CPU11
[    0.489949] GICv3: CPU11: found redistributor 501 region 0:0x0000000006360000
[    0.489959] GICv3: CPU11: using reserved LPI pending table @0x000000277fec0000
[    0.490070] CPU11: Booted secondary processor 0x0000000501 [0x410fd083]
[    0.521964] Detected PIPT I-cache on CPU12
[    0.521986] GICv3: CPU12: found redistributor 600 region 0:0x0000000006380000
[    0.521999] GICv3: CPU12: using reserved LPI pending table @0x000000277fed0000
[    0.522115] CPU12: Booted secondary processor 0x0000000600 [0x410fd083]
[    0.553987] Detected PIPT I-cache on CPU13
[    0.554005] GICv3: CPU13: found redistributor 601 region 0:0x00000000063a0000
[    0.554016] GICv3: CPU13: using reserved LPI pending table @0x000000277fee0000
[    0.554128] CPU13: Booted secondary processor 0x0000000601 [0x410fd083]
[    0.586019] Detected PIPT I-cache on CPU14
[    0.586042] GICv3: CPU14: found redistributor 700 region 0:0x00000000063c0000
[    0.586054] GICv3: CPU14: using reserved LPI pending table @0x000000277fef0000
[    0.586169] CPU14: Booted secondary processor 0x0000000700 [0x410fd083]
[    0.618040] Detected PIPT I-cache on CPU15
[    0.618060] GICv3: CPU15: found redistributor 701 region 0:0x00000000063e0000
[    0.618070] GICv3: CPU15: using reserved LPI pending table @0x000000277ff00000
[    0.618182] CPU15: Booted secondary processor 0x0000000701 [0x410fd083]
[    0.618222] smp: Brought up 1 node, 16 CPUs
[    0.998967] SMP: Total of 16 processors activated.
[    1.003838] CPU features: detected: 32-bit EL0 Support
[    1.009039] CPU features: detected: CRC32 instructions
[    1.025792] CPU: All CPU(s) started at EL2
[    1.029959] alternatives: patching kernel code
[    1.035161] devtmpfs: initialized
[    1.040783] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.050641] futex hash table entries: 4096 (order: 6, 262144 bytes, linear)
[    1.060228] xor: measuring software checksum speed
[    1.105097]    8regs     :  8463.000 MB/sec
[    1.149337]    32regs    :  8535.000 MB/sec
[    1.193579]    arm64_neon:  8465.000 MB/sec
[    1.197802] xor: using function: 32regs (8535.000 MB/sec)
[    1.203261] pinctrl core: initialized pinctrl subsystem
[    1.208961] DMI not present or invalid.
[    1.212993] NET: Registered protocol family 16
[    1.218081] DMA: preallocated 256 KiB pool for atomic allocations
[    1.224244] audit: initializing netlink subsys (disabled)
[    1.229773] audit: type=2000 audit(1.028:1): state=initialized audit_enabled=0 res=1
[    1.237614] cpuidle: using governor menu
[    1.241723] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    1.249130] Serial: AMBA PL011 UART driver
[    1.253307] imx mu driver is registered.
[    1.257275] imx rpmsg driver is registered.
[    1.262345] Machine: NXP Layerscape LX2160ARDB
[    1.266835] SoC family: QorIQ LX2160A
[    1.270528] SoC ID: svr:0x87360020, Revision: 2.0
[    1.276389] 21c0000.serial: ttyAMA0 at MMIO 0x21c0000 (irq = 20, base_baud = 0) is a SBSA
[    1.284662] printk: console [ttyAMA0] enabled
[    1.284662] printk: console [ttyAMA0] enabled
[    1.293403] printk: bootconsole [pl11] disabled
[    1.293403] printk: bootconsole [pl11] disabled
[    1.302740] sbsa-uart 21d0000.serial: aliased and non-aliased serial devices found in device tree. Serial port enumeration may be unpredictable.
[    1.315685] 21d0000.serial: ttyAMA1 at MMIO 0x21d0000 (irq = 21, base_baud = 0) is a SBSA
[    1.327948] fsl_mc_bus 80c000000.fsl-mc: MC firmware version: 10.20.4
[    1.336977] fsl_mc_dprc dprc.1: DMA mask not set
[    1.347858] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    1.354557] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    1.361248] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    1.367939] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    1.375930] cryptd: max_cpu_qlen set to 1000
[    1.450679] raid6: neonx8   gen()  4835 MB/s
[    1.522982] raid6: neonx8   xor()  4174 MB/s
[    1.595283] raid6: neonx4   gen()  4095 MB/s
[    1.667579] raid6: neonx4   xor()  3864 MB/s
[    1.739886] raid6: neonx2   gen()  3203 MB/s
[    1.812183] raid6: neonx2   xor()  3645 MB/s
[    1.884503] raid6: neonx1   gen()  2453 MB/s
[    1.956806] raid6: neonx1   xor()  2744 MB/s
[    2.029118] raid6: int64x8  gen()  2179 MB/s
[    2.101439] raid6: int64x8  xor()  1791 MB/s
[    2.173470] raid6: int64x4  gen()  2553 MB/s
[    2.245494] raid6: int64x4  xor()  1808 MB/s
[    2.317529] raid6: int64x2  gen()  1683 MB/s
[    2.389552] raid6: int64x2  xor()  1632 MB/s
[    2.461588] raid6: int64x1  gen()  1229 MB/s
[    2.533625] raid6: int64x1  xor()  1105 MB/s
[    2.537884] raid6: using algorithm neonx8 gen() 4835 MB/s
[    2.543271] raid6: .... xor() 4174 MB/s, rmw enabled
[    2.548225] raid6: using neon recovery algorithm
[    2.553075] ACPI: Interpreter disabled.
[    2.557459] iommu: Default domain type: Passthrough
[    2.562477] vgaarb: loaded
[    2.565317] SCSI subsystem initialized
[    2.569210] usbcore: registered new interface driver usbfs
[    2.574700] usbcore: registered new interface driver hub
[    2.580044] usbcore: registered new device driver usb
[    2.585265] imx-i2c 2000000.i2c: can't get pinctrl, bus recovery not supported
[    2.592556] i2c i2c-0: IMX I2C adapter registered
[    2.597321] imx-i2c 2040000.i2c: can't get pinctrl, bus recovery not supported
[    2.604597] i2c i2c-1: IMX I2C adapter registered
[    2.609475] mc: Linux media interface: v0.10
[    2.613748] videodev: Linux video capture interface: v2.00
[    2.619249] pps_core: LinuxPPS API ver. 1 registered
[    2.624202] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    2.633326] PTP clock support registered
[    2.637276] EDAC MC: Ver: 3.0.0
[    2.640772] No BMan portals available!
[    2.644666] QMan: Allocated lookup table at (____ptrval____), entry count 65537
[    2.652035] No QMan portals available!
[    2.655869] No USDPAA memory, no 'fsl,usdpaa-mem' in device-tree
[    2.662029] FPGA manager framework
[    2.665455] Advanced Linux Sound Architecture Driver Initialized.
[    2.671959] clocksource: Switched to clocksource arch_sys_counter
[    2.678129] VFS: Disk quotas dquot_6.6.0
[    2.682073] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.689010] pnp: PnP ACPI: disabled
[    2.695048] thermal_sys: Registered thermal governor 'step_wise'
[    2.695050] thermal_sys: Registered thermal governor 'power_allocator'
[    2.701469] NET: Registered protocol family 2
[    2.712524] tcp_listen_portaddr_hash hash table entries: 16384 (order: 6, 262144 bytes, linear)
[    2.721377] TCP established hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    2.730253] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    2.738039] TCP: Hash tables configured (established 262144 bind 65536)
[    2.744727] UDP hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    2.751902] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    2.759635] NET: Registered protocol family 1
[    2.764174] RPC: Registered named UNIX socket transport module.
[    2.770089] RPC: Registered udp transport module.
[    2.774783] RPC: Registered tcp transport module.
[    2.779478] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    2.785911] PCI: CLS 0 bytes, default 64
[    2.790557] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    2.798835] kvm [1]: IPA Size Limit: 44bits
[    2.803376] kvm [1]: vgic-v2@c0e0000
[    2.806956] kvm [1]: GIC system register CPU interface enabled
[    2.812945] kvm [1]: vgic interrupt IRQ1
[    2.817027] kvm [1]: Hyp mode initialized successfully
[    2.825780] Initialise system trusted keyrings
[    2.830276] workingset: timestamp_bits=44 max_order=23 bucket_order=0
[    2.837143] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    2.843116] NFS: Registering the id_resolver key type
[    2.848167] Key type id_resolver registered
[    2.852340] Key type id_legacy registered
[    2.856344] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    2.863042] jffs2: version 2.2. (NAND) c 2001-2006 Red Hat, Inc.
[    2.869224] fuse: init (API version 7.31)
[    2.873313] 9p: Installing v9fs 9p2000 file system support
[    2.886225] Key type asymmetric registered
[    2.890314] Asymmetric key parser 'x509' registered
[    2.895194] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    2.902645] io scheduler mq-deadline registered
[    2.907168] io scheduler kyber registered
[    2.914408] layerscape-pcie 3600000.pcie: host bridge /soc/pcie@3600000 ranges:
[    2.921720] layerscape-pcie 3600000.pcie:   MEM 0x9040000000..0x907fffffff -> 0x40000000
[    2.929884] layerscape-pcie 3600000.pcie: PCI host bridge to bus 0000:00
[    2.936577] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.942054] pci_bus 0000:00: root bus resource [mem 0x9040000000-0x907fffffff] (bus address [0x40000000-0x7fffffff])
[    2.952577] pci 0000:00:00.0: [1957:8d80] type 01 class 0x060400
[    2.958590] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    2.964852] pci 0000:00:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
[    2.971113] pci 0000:00:00.0: reg 0x38: [mem 0x9048000000-0x9048ffffff pref]
[    2.978193] pci 0000:00:00.0: supports D1 D2
[    2.982454] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    2.989541] pci 0000:00:00.0: BAR 1: assigned [mem 0x9040000000-0x9043ffffff]
[    2.996670] pci 0000:00:00.0: BAR 0: assigned [mem 0x9044000000-0x9044ffffff]
[    3.003801] pci 0000:00:00.0: BAR 6: assigned [mem 0x9045000000-0x9045ffffff pref]
[    3.011362] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    3.016591] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
[    3.025096] layerscape-pcie 3800000.pcie: host bridge /soc/pcie@3800000 ranges:
[    3.032403] layerscape-pcie 3800000.pcie:   MEM 0xa040000000..0xa07fffffff -> 0x40000000
[    3.040556] layerscape-pcie 3800000.pcie: PCI host bridge to bus 0001:00
[    3.047249] pci_bus 0001:00: root bus resource [bus 00-ff]
[    3.052725] pci_bus 0001:00: root bus resource [mem 0xa040000000-0xa07fffffff] (bus address [0x40000000-0x7fffffff])
[    3.063245] pci 0001:00:00.0: [1957:8d80] type 01 class 0x060400
[    3.069256] pci 0001:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[    3.075518] pci 0001:00:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
[    3.081780] pci 0001:00:00.0: reg 0x38: [mem 0xa048000000-0xa048ffffff pref]
[    3.088864] pci 0001:00:00.0: supports D1 D2
[    3.093125] pci 0001:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.099838] pci 0001:01:00.0: [10b5:8724] type 01 class 0x060400
[    3.106151] pci 0001:01:00.0: reg 0x10: [mem 0xa049000000-0xa04903ffff]
[    3.114004] pci 0001:01:00.0: PME# supported from D0 D3hot D3cold
[    3.121836] SError Interrupt on CPU12, code 0xbf000002 -- SError
[    3.121838] CPU: 12 PID: 1 Comm: swapper/0 Not tainted 5.4.3 #1
[    3.121838] Hardware name: NXP Layerscape LX2160ARDB (DT)
[    3.121839] pstate: 60000085 (nZCv daIf -PAN -UAO)
[    3.121840] pc : dw_pcie_read+0x80/0xa0
[    3.121840] lr : dw_pcie_access_other_conf.isra.16+0xd0/0xe8
[    3.121841] sp : ffff80001009b660
[    3.121842] x29: ffff80001009b660 x28: ffff80001009b954
[    3.121844] x27: ffff0026d9120800 x26: 0000000000000000
[    3.121845] x25: ffff800010045800 x24: 0000000000000000
[    3.121847] x23: ffff80001009b754 x22: 0000000000000004
[    3.121848] x21: ffff0026d911ac80 x20: 0000000000000000
[    3.121849] x19: ffff0026d911aca8 x18: ffffffffffffffff
[    3.121851] x17: 00000000f8fc8823 x16: 000000008812b067
[    3.121852] x15: ffffa3c7d2d29908 x14: ffff0026d912491c
[    3.121854] x13: ffff0026d91241a4 x12: 0000000000000038
[    3.121855] x11: 0000000000000004 x10: 0101010101010101
[    3.121857] x9 : 00000000872410b5 x8 : 0000000000000908
[    3.121858] x7 : 0000000000000908 x6 : ffff800010f00000
[    3.121860] x5 : 0000000000000000 x4 : ffff0026d911ac80
[    3.121861] x3 : 0000000000000003 x2 : ffff80001009b754
[    3.121862] x1 : 0000000000000000 x0 : ffff800010045800
[    3.121864] Kernel panic - not syncing: Asynchronous SError Interrupt
[    3.121865] CPU: 12 PID: 1 Comm: swapper/0 Not tainted 5.4.3 #1
[    3.121866] Hardware name: NXP Layerscape LX2160ARDB (DT)
[    3.121866] Call trace:
[    3.121867]  dump_backtrace+0x0/0x150
[    3.121867]  show_stack+0x14/0x20
[    3.121868]  dump_stack+0xbc/0x100
[    3.121869]  panic+0x16c/0x37c
[    3.121869]  __stack_chk_fail+0x0/0x18
[    3.121870]  arm64_serror_panic+0x74/0x88
[    3.121870]  do_serror+0x70/0x138
[    3.121871]  el1_error+0x84/0xf8
[    3.121872]  dw_pcie_read+0x80/0xa0
[    3.121872]  dw_pcie_rd_conf+0x120/0x128
[    3.121873]  pci_bus_read_config_dword+0x84/0xe0
[    3.121874]  pci_bus_generic_read_dev_vendor_id+0x2c/0x1a0
[    3.121874]  pci_bus_read_dev_vendor_id+0x48/0x68
[    3.121875]  pci_scan_single_device+0x78/0xf8
[    3.121875]  pci_scan_slot+0x34/0x128
[    3.121876]  pci_scan_child_bus_extend+0x50/0x290
[    3.121877]  pci_scan_child_bus+0x10/0x18
[    3.121877]  pci_scan_bridge_extend+0x228/0x560
[    3.121878]  pci_scan_child_bus_extend+0x12c/0x290
[    3.121879]  pci_scan_child_bus+0x10/0x18
[    3.121879]  pci_scan_bridge_extend+0x228/0x560
[    3.121880]  pci_scan_child_bus_extend+0x12c/0x290
[    3.121881]  pci_scan_root_bus_bridge+0x58/0xe8
[    3.121881]  dw_pcie_host_init+0x2fc/0x5e8
[    3.121882]  ls_pcie_probe+0x100/0x134
[    3.121883]  platform_drv_probe+0x50/0xa0
[    3.121883]  really_probe+0x108/0x348
[    3.121884]  driver_probe_device+0x58/0x100
[    3.121885]  device_driver_attach+0x6c/0x90
[    3.121885]  __driver_attach+0x84/0xc8
[    3.121886]  bus_for_each_dev+0x74/0xc8
[    3.121886]  driver_attach+0x20/0x28
[    3.121887]  bus_add_driver+0x148/0x1f0
[    3.121888]  driver_register+0x60/0x110
[    3.121888]  __platform_driver_probe+0x74/0x130
[    3.121889]  ls_pcie_driver_init+0x20/0x28
[    3.121890]  do_one_initcall+0x5c/0x1b0
[    3.121890]  kernel_init_freeable+0x1a4/0x24c
[    3.121891]  kernel_init+0x10/0x108
[    3.121891]  ret_from_fork+0x10/0x18
[    3.121908] SMP: stopping secondary CPUs
[    3.121909] Kernel Offset: 0x23c7c0c00000 from 0xffff800010000000
[    3.121910] PHYS_OFFSET: 0xfffff284c0000000
[    3.121910] CPU features: 0x0002,21806008
[    3.121911] Memory Limit: none

And here is the log when PCIe card is installed on other device which can run the card nomally.

(U-Boot)
other device# pci
Scanning PCI devices on bus 0
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x10b5     0x8724     Bridge device           0x04

(Userspace)
~ # lspci -nn
0001:00:00.0 PCI bridge [0604]: Device [xxxx:xxxx] 
0001:01:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8724] (rev ca)
0001:02:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8724] (rev ca)
0001:02:01.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8724] (rev ca)
0001:02:08.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8724] (rev ca)
0001:03:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
0001:03:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
0001:04:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
0001:04:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)

[ufedor]

Which NXP LSDK is used on the LX2160ARDB?

Please provide U-Boot booting log.

Please provide logs as attachments - not inline.

[tsungmin_wang]

LSDK: LSDK-20.04-V5.4-update-290520

And U-Boot is at attachments.

Thanks.

[ufedor]

What is the LX2160ARDB SW2[1] setting?

[tsungmin_wang]

SW2[1] is off.

[ufedor]

It seems the EP card is a combo device involving a Broadcom/PLX PEX8724 6-port PCIe Switch with 4 Intel 82599ES 10G SFI/SFP+ NIC connected underneath the PLX downstream port.

Please confirm.

[tsungmin_wang]

Yes, Intel 82599ES 10G SFI/SFP+ NIC is connected to the PLX downstream port.

[ufedor · Accepted solution]

Please try the patches from the attached ZIP package on LSDK 20.04 and let us know if the issue is resolved.

[tsungmin_wang]

Hi,

The problem is solved. After I enabling the driver IXGBE, the pcie card works.

Thanks for your help.

root@localhost:~# lspci
0000:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 8d80 (rev 20)
0001:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 8d80 (rev 20)
0001:01:00.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca)
0001:02:00.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca)
0001:02:01.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca)
0001:02:08.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca)
0001:03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
0001:03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
0001:04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
0001:04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
root@localhost:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enP1p3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:90:0b:45:4e:3c brd ff:ff:ff:ff:ff:ff
3: enP1p3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:90:0b:45:4e:3d brd ff:ff:ff:ff:ff:ff
4: enP1p4s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:90:0b:45:4e:3e brd ff:ff:ff:ff:ff:ff
5: enP1p4s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:90:0b:45:4e:3f brd ff:ff:ff:ff:ff:ff
6: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:0c:4b:aa:90 brd ff:ff:ff:ff:ff:ff

[tsungmin_wang]

Hi,

I'm still testing for this.

I have one sd card, it has been patched.

It can boot into userspace, and detect PCIe card successfully.

But the driver ixgbe is needed.

When I rebuild a linux with ixgbe enabled, it failed again.

I need some time to clear what the problem is, and thanks for your help.

If there is any update, I will reply.