P2020RDB-PCA, memory issue

[55573787]

Hi,

While bring up kernel Linux version 3.19.0 on board P2020RDB-PCA, it seems the memory has some problems.

The kernel reports:

Unable to handle kernel paging request for data at address 0xffffffff.

The whole u-boot information is :

++++++++++++++++ u-boot ++++++++++++++++++

U-Boot 2014.01 (Apr 15 2019 - 21:16:32)

CPU0: P2020, Version: 2.1, (0x80e20021)
Core: e500, Version: 5.1, (0x80211051)
Clock Configuration:
CPU0:1056 MHz, CPU1:1056 MHz,
CCB:528 MHz,
DDR:132 MHz (264 MT/s data rate) (Asynchronous), LBC:33 MHz
L1: D-cache 32 KiB enabled
I-cache 32 KiB enabled
Board: P2020RDB-PCA CPLD: V15.15 PCBA: V15.0
Error reading i2c boot information!
I2C: ready
SPI: ready
DRAM: DDR: failed to read SPD from address 82
SPD error on controller 0! Trying fallback to raw timing calculation
Detected UDIMM Fixed DDR on board
The choosen cas latency 5 is too large
2 GiB (DDR3, 64-bit, CL=5, ECC off)
Flash: 16 MiB
L2: 512 KiB enabled
NAND: ONFI param page 0 valid
8192 MiB
MMC: FSL_SDHC: 0
PCIe1: Root Complex of mini PCIe SLOT, no link, regs @ 0xffe0a000
PCIe1: Bus 00 - 00
PCIe2: disabled
In: serial
Out: serial
Err: serial
Net: eTSEC1 [PRIME], eTSEC2, eTSEC3
Hit 'CTRL + p' to stop autoboot: 0

+++++++++++++ kernel info ++++++++++++++++

=> bootm 1000000 2000000 c00000
WARNING: adjusting available memory to 30000000
## Booting kernel from Legacy Image at 01000000 ...
Image Name: Linux-3.19.0
Created: 2019-04-24 18:05:07 UTC
Image Type: PowerPC Linux Kernel Image (gzip compressed)
Data Size: 4371851 Bytes = 4.2 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 02000000 ...
Image Name: WiserMind(MPC) ramdisk image
Created: 2011-11-24 1:25:47 UTC
Image Type: PowerPC Linux RAMDisk Image (gzip compressed)
Data Size: 3447423 Bytes = 3.3 MiB
Load Address: 08000000
Entry Point: 08000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 00c00000
Booting using the fdt blob at 0xc00000
Uncompressing Kernel Image ... OK
Loading Ramdisk to 2fcb6000, end 2ffffa7f ... OK
Loading Device Tree to 03ff9000, end 03fff3f9 ... OK
WARNING: could not find compatible node fsl-usb2-dr: FDT_ERR_NOTFOUND.
Using P2020 RDB machine description
Memory CAM mapping: 256/256/256 Mb, residual: 1280Mb
Linux version 3.19.0 (dzq@ubuntu) (gcc version 4.3.2 (GCC) ) #3 SMP Thu Apr 25 02:04:49 CST 2019
Found initrd at 0xefcb6000:0xeffffa7f
CPU maps initialized for 1 thread per core
bootconsole [udbg0] enabled
setup_arch: initmem
mpc85xx_rdb_setup_arch()
mpc85xx_qe_init: Could not find Quicc Engine node
MPC85xx RDB board from Freescale Semiconductor
arch: exit
Zone ranges:
DMA [mem 0x00000000-0x2fffffff]
Normal empty
HighMem [mem 0x30000000-0x7fffffff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x00000000-0x7fffffff]
Initmem setup node 0 [mem 0x00000000-0x7fffffff]
MMU: Allocated 1088 bytes of context maps for 255 contexts
PERCPU: Embedded 9 pages/cpu @eec79000 s6368 r8192 d22304 u36864
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 522752
Kernel command line: root=/dev/ram rw console=ttyS0,115200
dzq:para=root,val=/dev/ram
dzq:para=rw,val=(null)
dzq:para=console,val=ttyS0,115200
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Sorting __ex_table...
Memory: 2001292K/2097152K available (6396K kernel code, 316K rwdata, 1880K rodata, 304K init, 196K bss, 95860K reserved, 0K cma-reserved, 1310712K highmem)
Kernel virtual memory layout:
* 0xfff5f000..0xfffff000 : fixmap
* 0xffc00000..0xffe00000 : highmem PTEs
* 0xffbfd000..0xffc00000 : early ioremap
* 0xf1000000..0xffbfd000 : vmalloc & ioremap
SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Hierarchical RCU implementation.
RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
NR_IRQS:512 nr_irqs:512 16
mpic: Setting up MPIC " OpenPIC " version 1.2 at ffe40000, max 2 CPUs
mpic: ISU size: 256, shift: 8, mask: ff
dzq:arch/powerpc/platforms/85xx/mpc85xx_rdb.c-67, no dtb resouce:fsl,MPC85XXRDB-CAMP
mpic: Initializing for 256 sources
mpc85xx_rdb_pic_init: Could not find qe-ic node
clocksource: timebase mult[f26c9b2] shift[24] registered
Console: colour dummy device 80x25
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
mpic: requesting IPIs...
Unable to handle kernel paging request for data at address 0xffffffff
Faulting instruction address: 0xc00fee68
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=8 P2020 RDB
Modules linked in:
CPU: 0 PID: 2 Comm: kthreadd Not tainted 3.19.0 #3
task: ea058600 ti: ea062000 task.ti: ea062000
NIP: c00fee68 LR: c005a684 CTR: 00000000
REGS: ea063d20 TRAP: 0300 Not tainted (3.19.0)
MSR: 00029000 <CE,EE,ME> CR: 82222028 XER: 00000000
DEAR: ffffffff ESR: 00000000
GPR00: 00000000 ea063dd0 ea058600 c08b17e8 c08b17e8 00000000 ea062000 eec7c930
GPR08: ffffffff c0864930 00000020 2e418000 22222024 00000000 c005d0b4 00000000
GPR16: 00000000 c08b0000 c086b96c 00000000 ea003200 c005cfdc 00000000 00000000
GPR24: 00000000 00000000 c08c0000 c086b9d4 00000000 000000d0 ea001d80 ffffffff
NIP [c00fee68] kmem_cache_alloc+0x58/0x148
LR [c005a684] alloc_pid+0x28/0x428
Call Trace:
[ea063dd0] [c08620a0] runqueues+0x0/0x540 (unreliable)
[ea063df0] [c005a684] alloc_pid+0x28/0x428
[ea063e50] [c003f910] copy_process+0x5d0/0x11d4
[ea063ed0] [c0040590] do_fork+0x7c/0x2c0
[ea063f10] [c005d180] kthreadd+0xcc/0x144
[ea063f40] [c000ed7c] ret_from_kernel_thread+0x5c/0x64
Instruction dump:
7c832378 54260024 80060008 813e0000 5400103a 7d64002e 7ce95a14 81470004
7fe9582e 2f9f0000 419e00dc 801e0014 <7cff002e> 7ca000a6 7c000146 80060008
---[ end trace 1dbe0ab83a4b57b7 ]---

[ufedor]

Which exactly board is in question?

Please provide clear photos of all on-board inscriptions and stickers.

Which NXP SDK is used?

In the log:

> DDR:132 MHz

Is this as projected?

> Board: P2020RDB-PCA CPLD: V15.15 PCBA: V15.0

Illegal versions of CPLD and PCBA - why?

> Error reading i2c boot information!

> DRAM: DDR: failed to read SPD from address 82

Why is this?

[55573787]

Hi ufedor,

      This is uncustom P2020RDB-PC board. We designed new board based on 

      The main modification is changing DDR size from 1G to 2G.

       U-Boot configuration:

                make distclean;make P2020RDB-PC_config;make

       Kernel configuration:

                make mpc85xx_smp_defconfig

Thanks. BR.

[ufedor]

1) Which NXP SDK is used?

2) In the log:

> DDR:132 MHz

Is this as projected?

3) Provide complete board schematics as searchable PDF for inspection.

[ufedor]

1) What is the measured MCK2 frequency?

2) Is the fly-by topology implemented according to the AN3940 - Hardware and Layout Design Considerations for DDR3 SDRAM?

3) The custom board schematics significantly differs from the P2020-RDB - it does not contain I2C EEPROM which is used on the P2020-RDB to store SPD data for the on-board DDR3 SDRAM. If the SPD I2C is absent it is recommended to use the CodeWarrior QCVS DDR Tool to evaluate and optimize P2020 DDR controller settings for the specific configuration and, after that, insert them into the U-Boot code.

[55573787]

Hi 

      We used the QCVS Tool to adjust the parameters for DDR configuration.

      Now the u-Boot logs:

+++++++++++++++++++++

U-Boot 2014.01 (Apr 28 2019 - 14:14:04)

CPU0: P2020, Version: 2.1, (0x80e20021)
Core: e500, Version: 5.1, (0x80211051)
Clock Configuration:
CPU0:1056 MHz, CPU1:1056 MHz,
CCB:528 MHz,
DDR:264 MHz (528 MT/s data rate) (Asynchronous), LBC:33 MHz
L1: D-cache 32 KiB enabled
I-cache 32 KiB enabled
Board: P2020RDB-PCA CPLD: V15.15 PCBA: V15.0
Error reading i2c boot information!
I2C: ready
SPI: ready
DRAM: Configuring DDR for 528 MT/s data rate
2 GiB (DDR3, 64-bit, CL=6, ECC on)
Flash: 16 MiB
L2: 512 KiB enabled

+++++++++++++++++++++++++++++++

              But the kernel still couldn't be booted up. It failed at:

+++++++++++++++++++++++++++++++

clocksource: timebase mult[f26c9b2] shift[24] registered
Console: colour dummy device 80x25
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
Machine check in kernel mode.
Caused by (from MCSR=8): Bus - Read Data Bus Error
Oops: Machine check, sig: 7 [#1]
SMP NR_CPUS=8 P2020 RDB
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.0 #10
task: c0868300 ti: efff0000 task.ti: c08a6000
NIP: c00177f0 LR: c011fe60 CTR: 00000000
REGS: efff1f10 TRAP: 0204 Not tainted (3.19.0)
MSR: 00021000 <CE,ME> CR: 24242028 XER: 00000000
DEAR: f100d000 ESR: 00800000
GPR00: 00000054 c08a7d30 c0868300 e9822a90 00000000 00000150 e9822bdc c08b17e8
GPR08: 00000000 e9822a64 00000001 e9822a40 24242022 08807621 c08a6000 ea00e570
GPR16: 00000000 c013ab7c 00000000 7ffa9c0c 0142b5d0 ffffffff 000000d0 00000000
GPR24: 00000000 00000000 e9822000 80010400 00000000 e9822a90 00000006 e9822a00
NIP [c00177f0] memset+0x3c/0x5c
LR [c011fe60] inode_init_once+0x24/0x74
Call Trace:
[c08a7d30] [c011fe94] inode_init_once+0x58/0x74 (unreliable)
[c08a7d50] [c013b394] init_once+0x6c/0x94
[c08a7d70] [c00fe3b4] new_slab+0x1a8/0x2d0
[c08a7da0] [c00fe6d8] __slab_alloc+0xf0/0x530
[c08a7e70] [c00fef50] kmem_cache_alloc+0x140/0x148
[c08a7e90] [c013ab7c] bdev_alloc_inode+0x1c/0x3c
[c08a7ea0] [c0120cf4] alloc_inode+0x2c/0xbc
[c08a7eb0] [c01210cc] new_inode_pseudo+0x14/0x58
[c08a7ec0] [c0121134] new_inode+0x24/0x48
[c08a7ed0] [c012d784] mount_pseudo+0xb0/0x17c
[c08a7f20] [c013b200] bd_mount+0x30/0x40
[c08a7f30] [c0106ec0] mount_fs+0x20/0xd8
[c08a7f50] [c0127b3c] vfs_kern_mount+0x60/0x120
[c08a7f70] [c0127c18] kern_mount_data+0x1c/0x40
[c08a7f80] [c08304b8] bdev_cache_init+0x68/0xb8
[c08a7f90] [c082fc7c] vfs_caches_init+0xb4/0x140
[c08a7fb0] [c0819b04] start_kernel+0x33c/0x3c4
[c08a7ff0] [c00003d8] set_ivor+0x140/0x17c
Instruction dump:
28050004 41800030 94860004 4d820020 70c00003 7ca02a14 7cc03050 54a0f0be
7c0903a6 4240000c 94860004 4200fffc <70a50003> 2c050000 4d820020 7ca903a6
---[ end trace dc8fa200cb88537f ]---

   Attached files are two uboot&kernel logs and the kernel configuration file.

    The kernel is from QorIQ-SDK-V1.9-SOURCE-20160310-yocto.

    Would you please check the logs and give us some advice?

   Also, would you provided the standard kernel configuration for reference.

    Thanks.

[ufedor · Accepted solution]

Please confirm that all QCVS DDR Tool tests passed, the DDR controller settings were optimized, exported and inserted into U-Boot code.

[55573787]

Hi  ufedor

       Our hardware engineer are going to check the DDR.

       For the software parts(Kernel and dts), as we didn't have the P2020RDB reference board, would you boot up the reference board by our uImage and DTS attached and provide the log?  So, we could check whether the kernel configuration and DTS are right configured.

       Or, would you mind to provide the uImage and DTS on P2020RDB reference board  to us? So we could check our hardware from another side.

        Thanks for your help.

Derek  Dou

[ufedor]

Sorry, we do not have such board at hand.