Issue with 2 PCIE controllers enabled

rksvy · ‎12-07-2013

Fellow engineers,

I am working with an p4080 based board. I am trying to get 2 PCIE controllers probed properly.

In uboot I have no problems scanning and discovering what is connected to both controllers/PCI bridges.

For both PCIE1/2 uboot sets up the Primary, secondary and Subordinate bus numbers to 0,1,1 respectively.

When linux boots up and probes the controllers, PCIE1 is probed and the bridge scanned properly but PCIE2 is probed at the bridge but not attempted a scan.

I see this message

"pci 0001:02:00.0: bridge configuration invalid ([bus 01-01]), reconfiguring

"

I updated uboot to set the secondary and subordinate numbers to 2 (left the primary number to 0) and a subsequent kernel boot scanned the bus for PCIE2 successfully.

I found these numbers to be very critical since the device tree blob (bus-range) for pci is also based off these.

I'd like to get a good fix rather than the uboot hack and get better understanding of the problem. If there are any pointers someone could provide it would be awesome.

Thank you

Sivakumar · ‎06-14-2014

Hi Ruchika,

I am also facing the same problem.

In mine, pci-1 is not detecting the device(sata controller), bcz of sata controller problem.

pci-2 is detecting the device(xr17v35x uart controller) in the uboot. And when booting the kernel i am seeing the exact message that u see.

"pci 0001:02:00.0: Primary bus is hard wired to 0

pci 0001:02:00.0: bridge configuration invalid ([bus 01-01]), reconfiguring"

After kernel booted, the driver is loading successfully.But an attempt to read/write to the devices make the kernel crash

root@p1023:~# echo 123 > /dev/ttyXR0

Machine check in kernel mode.

Caused by (from MCSR=10008): Bus - Read Data Bus Error

Oops: Machine check, sig: 7 [#1]

P1023 RDB

After that i updated the uboot(in pciauto_postscan_setup_bridge() to secondary and subordinate bus number to 2), since it is fixed the bridge reconfiguring issue for u.

Now uboot is detecting the device as before while it is booting. But when i list it via command "pci 2 long", I am not seeing the device and checked the "pci 1 long' that second and subordinate bus numbers are 2. But subsequent kernel boot now shows as below, and hence i am seeing the same kernel crash, when an attempt to read/write to the device.

"pci 0001:02:00.0: Primary bus is hard wired to 0

pci 0001:02:00.0: bridge configuration invalid ([bus 02-02]), reconfiguring"

So, can you plz let me know exactly what u changed in the uboot and/or dts file to fix this issue.?

Thanking you

S.Sivakumar

yipingwang · ‎06-17-2014

Which version SDK are you using?

Would you please provide the whole u-boot and Kernel boot up log(without your modification)? We will do more investigation.

Sivakumar · ‎06-17-2014

Hi,

We have fixed the issue here.

We had mapped the PCIe2 in the 36 bit memory space. But the driver(exar xr17v35x) expects the device to be within the 32bit memory space.

Then we changed it back, only the PCIe2 to be within the 32bit space in the uboot/law and dts file.. This is working now(crashing is not happening)

Thanks for the support

I am using freescale SDK 1.4 / kernel 3.8.13.

Thanks & Regards,

S.Sivakumar

gokulcg · ‎08-13-2014

I am facing same problem with PCIE driver in new Linux kernel

shipped along with SDK-V1.5-20131219 .I was using old

kernel Linux version 3.0.48 previously and now updated to

Linux version 3.8.13-rt9 inorder to add IEEE1588 support in

Eatherner driver . After updating to the new kernel ,the device

drivers are not woring properly and i am getting some error

messages in the boot-up . Please find below bootlog snippet

for both old kernel (PCIe working ) and the new one (PCIe not

working )

Working old kernel(linux version 3.0.48)

===============================

PCIe1: disabled

PCIe2: Root Complex, x1, regs @ 0xfe201000

01:00.0 - 13a8:0358 - Simple comm. controller

PCIe2: Bus 00 - 01

PCIe3: disabled

In: serial

Out: serial

Err: serial

SERDES bank 1 reference clock: 100MHz

SERDES bank 2 reference clock: 100MHz

Net: Initializing Fman

Fman1: Uploading microcode version 106.1.4

Change supported feature to 100:FD/HD for addr 6 features(X) 28C

FM1@DTSEC1 connected to Broadcom BCM54616S, address 6

Fiber optic mode active

FM1@DTSEC2 connected to Broadcom BCM54616S, address 7

Change supported feature to 100:FD/HD for addr 12 features(X) 28C

FM1@DTSEC3 connected to Broadcom BCM54616S, address 12

Fiber optic mode active

FM1@DTSEC4 connected to Broadcom BCM54616S, address 9

FM1@DTSEC5 connected to Broadcom BCM54616S, address 10

FM1@DTSEC1, FM1@DTSEC2, FM1@DTSEC3, FM1@DTSEC4, FM1@DTSEC5

Hit any key to stop autoboot: 0

=> boot

Setting bus to 1

Error writing the chip.

NAND read: device 0 offset 0x100000, size 0x500000

5242880 bytes read: OK

NAND read: device 0 offset 0x600000, size 0x100000

1048576 bytes read: OK

WARNING: adjusting available memory to 30000000

## Booting kernel from Legacy Image at 01000000 ...

Image Name: Linux-3.0.48-rt70

Image Type: PowerPC Linux Kernel Image (gzip compressed)

Data Size: 3972344 Bytes = 3.8 MiB

Load Address: 00000000

Entry Point: 00000000

Verifying Checksum ... OK

## Flattened Device Tree blob at 00c00000

Booting using the fdt blob at 0x00c00000

Uncompressing Kernel Image ... OK

Loading Device Tree to 03fe5000, end 03fff1a9 ... OK

WARNING: could not find compatible node fsl-usb2-mph or fsl-usb2-dr: FDT_ERR_NOTFOUND.

Using P2041 RDB machine description

Memory CAM mapping: 256/256/256/256 Mb, residual: 0Mb

Linux version 3.0.48-rt70 (root@localhost.localdomain) (gcc version 4.6.2 (GCC) ) #138 SMP PREEMPT RT Fri May 30 17:51:01 IST 2014

No /soc@ffe000000/qman@318000 property 'fsl,qman-fqd', using memblock_alloc(0000000000200000)

No /soc@ffe000000/qman@318000 property 'fsl,qman-pfdr', using memblock_alloc(0000000001000000)

Qman ver:0a01,01,02

No /soc@ffe000000/bman@31a000 property 'fsl,bman-fbpr', using memblock_alloc(0000000001000000)

Bman ver:0a02,01,00

pme: No /soc@ffe000000/pme@316000 property 'fsl,pme-pdsr', using memblock_alloc(0x0000000001000000)

pme: No /soc@ffe000000/pme@316000 property 'fsl,pme-sre', using memblock_alloc(0x0000000000a00000)

No USDPAA memory, no 'usdpaa_mem' bootarg

CPU maps initialized for 1 thread per core

bootconsole [udbg0] enabled

Found FSL PCI host bridge at 0x0000000ffe201000. Firmware bus number: 0->1

PCI host bridge /pcie@ffe201000 ranges:

MEM 0x0000000c20000000..0x0000000c3fffffff -> 0x00000000e0000000

IO 0x0000000ff8010000..0x0000000ff801ffff -> 0x0000000000000000

/pcie@ffe201000: PCICSRBAR @ 0xdf000000

P2041 RDB board from Freescale Semiconductor

Zone PFN ranges:

DMA 0x00000000 -> 0x00040000

Normal empty

Movable zone start PFN for each node

early_node_map[1] active PFN ranges

0: 0x00000000 -> 0x00040000

MMU: Allocated 1088 bytes of context maps for 255 contexts

PERCPU: Embedded 10 pages/cpu @b1083000 s18752 r8192 d14016 u40960

Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096

Kernel command line: root=/dev/mtdblock4 rootfstype=jffs2 rw console=ttyS0,115200

PID hash table entries: 4096 (order: 2, 16384 bytes)

Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)

Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)

Memory: 969004k/1048576k available (7824k kernel code, 79572k reserved, 280k data, 553k bss, 276k init)

Kernel virtual memory layout:

* 0xfffdf000..0xfffff000 : fixmap

* 0xfdfe9000..0xfe000000 : early ioremap

* 0xf1000000..0xfdfe9000 : vmalloc & ioremap

Preemptible hierarchical RCU implementation.

RCU debugfs-based tracing is enabled.

NR_IRQS:512 nr_irqs:512 16

mpic: Setting up MPIC " OpenPIC " version 1.2 at ffe040000, max 4 CPUs

mpic: ISU size: 256, shift: 8, mask: ff

mpic: Initializing for 256 sources

clocksource: timebase mult[efffffb] shift[22] registered

Console: colour dummy device 80x25

pid_max: default: 32768 minimum: 301

Mount-cache hash table entries: 512

mpic: requesting IPIs...

e500 family performance monitor hardware support registered

Brought up 4 CPUs

devtmpfs: initialized

xor: measuring software checksum speed

8regs : 760.000 MB/sec

8regs_prefetch: 660.000 MB/sec

32regs : 784.000 MB/sec

32regs_prefetch: 677.000 MB/sec

xor: using function: 32regs (784.000 MB/sec)

NET: Registered protocol family 16

EDAC PCI0: Giving out device to module 'MPC85xx_edac' controller 'mpc85xx_pci_err': DEV 'ffe201000.pcie' (INTERRUPT)

MPC85xx_edac acquired irq 493 for PCI Err

MPC85xx_edac PCI err registered

PCI: Probing PCI hardware

pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)

PCIE error(s) detected

pci 0000:00:00.0: PCI bridge to [bus 01-ff]

PCIE ERR_DR register: 0x80020000

PCIE ERR_CAP_STAT register: 0x80000001

PCIE ERR_CAP_R0 register: 0x00000800

PCIE ERR_CAP_R1 register: 0x00000000

PCIE ERR_CAP_R2 register: 0x00000000

PCIE ERR_CAP_R3 register: 0x00000000

pci 0000:00:00.0: PCI bridge to [bus 01-01]

pci 0000:00:00.0: bridge window [io 0xfdfea000-0xfdff9fff]

pci 0000:00:00.0: bridge window [mem 0xc20000000-0xc3fffffff]

pci 0000:00:00.0: bridge window [mem pref disabled]

bio: create slab <bio-0> at 0

raid6: int32x1 62 MB/s

raid6: int32x2 112 MB/s

raid6: int32x4 189 MB/s

raid6: int32x8 166 MB/s

raid6: using algorithm int32x4 (189 MB/s)

vgaarb: loaded

SCSI subsystem initialized

usbcore: registered new interface driver usbfs

usbcore: registered new interface driver hub

usbcore: registered new device driver usb

Not working new kernel(linux version 3.8.13)

===============================

PCIe1: disabled

PCIe2: Root Complex, x1, regs @ 0xfe201000

01:00.0 - 13a8:0358 - Simple comm. controller

PCIe2: Bus 00 - 01

PCIe3: disabled

In: serial

Out: serial

Err: serial

SERDES bank 1 reference clock: 100MHz

SERDES bank 2 reference clock: 100MHz

Net: Initializing Fman

Fman1: Uploading microcode version 106.1.4

Change supported feature to 100:FD/HD for addr 6 features(X) 28C

FM1@DTSEC1 connected to Broadcom BCM54616S, address 6

Fiber optic mode active

FM1@DTSEC2 connected to Broadcom BCM54616S, address 7

Change supported feature to 100:FD/HD for addr 12 features(X) 28C

FM1@DTSEC3 connected to Broadcom BCM54616S, address 12

Fiber optic mode active

FM1@DTSEC4 connected to Broadcom BCM54616S, address 9

FM1@DTSEC5 connected to Broadcom BCM54616S, address 10

FM1@DTSEC1, FM1@DTSEC2, FM1@DTSEC3, FM1@DTSEC4, FM1@DTSEC5

Hit any key to stop autoboot: 0

Setting bus to 1

Error writing the chip.

NAND read: device 0 offset 0x100000, size 0x500000

5242880 bytes read: OK

NAND read: device 0 offset 0x600000, size 0x100000

1048576 bytes read: OK

WARNING: adjusting available memory to 30000000

## Booting kernel from Legacy Image at 01000000 ...

Image Name: Linux-3.8.13-rt9

Image Type: PowerPC Linux Kernel Image (gzip compressed)

Data Size: 4542147 Bytes = 4.3 MiB

Load Address: 00000000

Entry Point: 00000000

Verifying Checksum ... OK

## Flattened Device Tree blob at 00c00000

Booting using the fdt blob at 0x00c00000

Uncompressing Kernel Image ... OK

Loading Device Tree to 03fe5000, end 03fff1a9 ... OK

WARNING: could not find compatible node fsl-usb2-mph or fsl-usb2-dr: FDT_ERR_NOTFOUND.

Using P2041 RDB machine description

Memory CAM mapping: 256/256/256/256 Mb, residual: 0Mb

Linux version 3.8.13-rt9 (root@localhost.localdomain) (gcc version 4.7.3 (GCC) ) #7 SMP PREEMPT RT Wed Aug 6 13:16:30 IST 201 4

No /soc@ffe000000/qman@318000 property 'fsl,qman-fqd', using memblock_alloc(0000000000200000)

No /soc@ffe000000/qman@318000 property 'fsl,qman-pfdr', using memblock_alloc(0000000002000000)

Qman ver:0a01,01,02

No /soc@ffe000000/bman@31a000 property 'fsl,bman-fbpr', using memblock_alloc(0000000001000000)

Bman ver:0a02,01,00

pme: No /soc@ffe000000/pme@316000 property 'fsl,pme-pdsr', using memblock_alloc(0x0000000001000000)

pme: No /soc@ffe000000/pme@316000 property 'fsl,pme-sre', using memblock_alloc(0x0000000000a00000)

No USDPAA memory, no 'usdpaa_mem' bootarg

CPU maps initialized for 1 thread per core

setup_arch: bootmem

arch: exit

bootconsole [udbg0] enabled

P2041 RDB board from Freescale Semiconductor

Zone ranges:

DMA [mem 0x00000000-0x3fffffff]

Normal empty

Movable zone start for each node

Early memory node ranges

node 0: [mem 0x00000000-0x3fffffff]

MMU: Allocated 1088 bytes of context maps for 255 contexts

PERCPU: Embedded 8 pages/cpu @b11de000 s10752 r8192 d13824 u32768

Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096

Kernel command line: root=/dev/mtdblock4 rootfstype=jffs2 rw console=ttyS0,115200

PID hash table entries: 4096 (order: 2, 16384 bytes)

Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)

Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)

Memory: 885312k/1048576k available (9156k kernel code, 163264k reserved, 340k data, 552k bss, 268k init)

Kernel virtual memory layout:

* 0xfffdf000..0xfffff000 : fixmap

* 0xfdffb000..0xfe000000 : early ioremap

* 0xf1000000..0xfdffb000 : vmalloc & ioremap

Preemptible hierarchical RCU implementation.

RCU debugfs-based tracing is enabled.

RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.

NR_IRQS:512 nr_irqs:512 16

mpic: Setting up MPIC " OpenPIC " version 1.2 at ffe040000, max 4 CPUs

mpic: ISU size: 512, shift: 9, mask: 1ff

mpic: Initializing for 512 sources

clocksource: timebase mult[3bffffec] shift[24] registered

Console: colour dummy device 80x25

pid_max: default: 32768 minimum: 301

Mount-cache hash table entries: 512

mpic: requesting IPIs...

e500 family performance monitor hardware support registered

Brought up 4 CPUs

devtmpfs: initialized

xor: measuring software checksum speed

8regs : 762.000 MB/sec

8regs_prefetch: 662.000 MB/sec

32regs : 820.000 MB/sec

32regs_prefetch: 727.000 MB/sec

xor: using function: 32regs (820.000 MB/sec)

NET: Registered protocol family 16

Found FSL PCI host bridge at 0x0000000ffe201000. Firmware bus number: 0->1

PCI host bridge /pcie@ffe201000 (primary) ranges:

MEM 0x0000000c20000000..0x0000000c3fffffff -> 0x00000000e0000000

IO 0x0000000ff8010000..0x0000000ff801ffff -> 0x0000000000000000

/pcie@ffe201000: PCICSRBAR @ 0xdf000000

EDAC PCI0: Giving out device to module 'MPC85xx_edac' controller 'mpc85xx_pci_err': DEV 'ffe201000.pcie' (INTERRUPT)

MPC85xx_edac acquired irq 481 for PCI Err

MPC85xx_edac PCI err registered

PCI: Probing PCI hardware

fsl-pci ffe201000.pcie: PCI host bridge to bus 0000:00

pci_bus 0000:00: root bus resource [io 0x0000-0xffff]

pci_bus 0000:00: root bus resource [mem 0xc20000000-0xc3fffffff] (bus address [0xe0000000-0xffffffff])

pci_bus 0000:00: root bus resource [bus 00-ff]

PCIE error(s) detected

PCIE ERR_DR register: 0x80020000

PCIE ERR_CAP_STAT register: 0x80000001

PCIE ERR_CAP_R0 register: 0x00000800

pci 0000:00:00.0: PCI bridge to [bus 01-ff]

PCIE ERR_CAP_R1 register: 0x00000000

PCIE ERR_CAP_R2 register: 0x00000000

PCIE ERR_CAP_R3 register: 0x00000000

PCI: Cannot allocate resource region 0 of device 0000:00:00.0, will remap

pci 0000:00:00.0: BAR 0: can't assign mem (size 0x1000000)

pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x200000)

pci 0000:00:00.0: PCI bridge to [bus 01]

pci 0000:00:00.0: bridge window [io 0x0000-0xffff]

pci 0000:00:00.0: bridge window [mem 0xc20000000-0xc3fffffff]

bio: create slab <bio-0> at 0

raid6: int32x1 164 MB/s

raid6: int32x2 227 MB/s

raid6: int32x4 299 MB/s

raid6: int32x8 217 MB/s

raid6: using algorithm int32x4 (299 MB/s)

raid6: using intx1 recovery algorithm

vgaarb: loaded

SCSI subsystem initialized

usbcore: registered new interface driver usbfs

Please let me know how to fix the same

I read following mail thread , but couldn't understand what is the problem .
linuxppc-dev - Re: PCIE device errors after linux kernel upgrade | Threaded View

scottwood · ‎08-13-2014

Please post your issue under a new thread -- it doesn't appear to be the same issue. Your log dump doesn't even seem to involve multiple PCIe controllers, as per the thread subject, and your working log shows a 36-bit physical address, so your problem isn't a driver that can't handle such addresses.

When you do create a new thread, be sure to indicate what's actually not working. Those error messages may look scary, but are not likely to be related to whatever problem you're having.