T4240RDB SDK 2.0 Kernel crashes

[sareenakp]

Hi,

We have a T4240RDB board. Currently there is a QorIQ SDK v1.5 U-Boot, fman-ucode and kernel V3.12(SDK 1.7) installed. We want to upgrade the linux to 4.1 of the SDK 2.0. However when using SDK 2.0 linux, we observe a crash

kernel BUG at /media/nslab12/work2/Yocto/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb/tmp/work-shared/t4240rdb/kernel-source/drivers/staging/fsl_qbman/qman_config.c:521!

Hence , understanding that SDK 2.0 requires fman-ucode 106.4.18 for correct functioning, we tried to install the fman ucode 106.4.18 of the SDK 2.0 package.  That led us to the problem as given in 

T4240: Cannot change to alternative flash boot section  (due to the flash map change between SDK V1.5 and SDK V2.0)

In order to quicky verify if upgrading fman ucode to 106.4.18 will solve the crash, I flashed the fman ucode to the fman ucode address as in SDK V1.5 FLASH MAP ( EFF40000)

Now we see that the new fman ucode is loaded. However the crash is still seen.

Can you please help me solving the crash?  Please provide me some directions.

The Kernel crash :

kernel BUG at /media/nslab12/work2/Yocto/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb/tmp/work-shared/t4240rdb/kernel-source/drivers/staging/fsl_qbman/qman_config.c:521!
Oops: Exception in kernel mode, sig: 5 [#1]
SMP NR_CPUS=24 CoreNet Generic
Modules linked in:
CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.1.8-rt8+gbd51baf #4
task: c00000017805cb10 ti: c0000001780a0000 task.ti: c0000001780a0000
NIP: c000000000af0ab8 LR: c000000000af0ab0 CTR: c00000000001eb78
REGS: c0000001780a3790 TRAP: 0700 Not tainted (4.1.8-rt8+gbd51baf)
MSR: 0000000080029000 <CE,EE,ME> CR: 24adbe22 XER: 20000000
SOFTE: 1
GPR00: c000000000af0ab0 c0000001780a3a10 c000000000c2c400 000000000000001f
GPR04: 0000000000000001 0000000000000000 0000000000000067 c000000000b8fa30
GPR08: 0000000000000000 fffffffffffffff4 c000000000b1f108 0000000000000006
GPR12: 0000000022adbe22 c00000003fff56c0 c0000000000021cc 0000000000000000
GPR16: 0000000000000000 0000000000000000 c000000000a40e00 c000000000a42d18
GPR20: c000000000ce6e88 c000000000a42c48 c000000000a42c70 c000000000a42c18
GPR24: c000000000a42c98 c000000000a40e70 c000000000a42ce8 c000000000a42cb8
GPR28: c000000000b56d38 c000000000ce6e80 c000000000a01420 c00000017fff9a00
NIP [c000000000af0ab8] .qman_init_early+0x1dc/0x41c
LR [c000000000af0ab0] .qman_init_early+0x1d4/0x41c
Call Trace:
[c0000001780a3a10] [c000000000af0ab0] .qman_init_early+0x1d4/0x41c (unreliable)
[c0000001780a3b70] [c00000000000197c] .do_one_initcall+0xb4/0x22c
[c0000001780a3c60] [c000000000ac1e28] .kernel_init_freeable+0x190/0x2a8
[c0000001780a3db0] [c0000000000021e8] .kernel_init+0x1c/0xe34
[c0000001780a3e30] [c000000000000998] .ret_from_kernel_thread+0x58/0xc0
Instruction dump:
41be0200 7ee3bb78 4bcd0011 60000000 39200000 e89d0008 7ea3ab78 e8bc0000
912100ac 4bccfff5 60000000 e92100ae <0b090000> e89d0010 39200000 e8bc0008
---[ end trace a1a9ca8513bef9d2 ]---

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005

Rebooting in 180 seconds..

______________________________________________________________________

The power on Log is as follows:

U-Boot 2013.01QorIQ-SDK-V1.5 (Jun 05 2014 - 16:58:13)

CPU0: T4240E, Version: 2.0, (0x82480020)
Core: E6500, Version: 2.0, (0x80400120)
Clock Configuration:
CPU0:1666.667 MHz, CPU1:1666.667 MHz, CPU2:1666.667 MHz, CPU3:1666.667 MHz,
CPU4:1666.667 MHz, CPU5:1666.667 MHz, CPU6:1666.667 MHz, CPU7:1666.667 MHz,
CPU8:1666.667 MHz, CPU9:1666.667 MHz, CPU10:1666.667 MHz, CPU11:1666.667 MHz,
CCB:733.333 MHz,
DDR:933.333 MHz (1866.667 MT/s data rate) (Asynchronous), IFC:183.333 MHz
FMAN1: 733.333 MHz
FMAN2: 733.333 MHz
QMAN: 366.667 MHz
PME: 533.333 MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled
Reset Configuration Word (RCW):
00000000: 16070019 18101916 00000000 00000000
00000010: 70701050 00448c00 0c020000 f5000000
00000020: 00000000 ee0000ee 00000000 000287fc
00000030: 00000000 50000000 00000000 00000028
Board: T4240RDB, SERDES Reference Clocks: SERDES1=100MHz SERDES2=156.25MHz SERDES3=100MHz SERDES4=100MHz
I2C: ready
SPI: ready
DRAM: Initializing....using SPD
Detected UDIMM 9JSF25672AZ-2G1K1
Detected UDIMM 9JSF25672AZ-2G1K1
Detected UDIMM 9JSF25672AZ-2G1K1
4 GiB left unmapped
DDR: 6 GiB (DDR3, 64-bit, CL=13, ECC on)
DDR Controller Interleaving Mode: 3-way 4KB
Flash: 128 MiB
L2: 2048 KB enabled
enable l2 for cluster 1 fec60000
enable l2 for cluster 2 feca0000
Corenet Platform Cache: 1536 KB enabled
Using SERDES1 Protocol: 28 (0x1c)
Using SERDES2 Protocol: 56 (0x38)
Using SERDES3 Protocol: 2 (0x2)
Using SERDES4 Protocol: 10 (0xa)
SRIO1: disabled
SRIO2: disabled
NAND: 2048 MiB
MMC: FSL_SDHC: 0
PCIe1: Root Complex, no link, regs @ 0xfe240000
PCIe1: Bus 00 - 00
PCIe2: disabled
PCIe3: Root Complex, no link, regs @ 0xfe260000
PCIe3: Bus 01 - 01
PCIe4: Root Complex, no link, regs @ 0xfe270000
PCIe4: Bus 02 - 02
In: serial
Out: serial
Err: serial
Warning: SERDES2 expects reference clock 125MHz, but actual is 156.25MHz
Net: Fman1: Uploading microcode version 106.4.18
Failed to connect
Failed to connect
cs4340_phy_init error.
cs4340_phy_init error.
Fman2: Uploading microcode version 106.4.18
cs4340_phy_init error.
cs4340_phy_init error.
FM1@DTSEC1 [PRIME], FM1@DTSEC2, FM1@DTSEC3, FM1@DTSEC4, FM1@TGEC1, FM1@TGEC2, FM2@DTSEC1, FM2@DTSEC2, FM2@DTSEC3, FM2@DTSEC4, FM2@TGEC1, FM2@TGEC2
Hit any key to stop autoboot: 0
=>
=>

=> setenv bootargs root=/dev/mmcblk0p1 rw rootfstype=ext2 rootwait rootdelay=2 console=$consoledev,$baudrate fsl_fm_max_frm=9600;tftp $loadaddr $bootfile;tftp e00000 $fdtfile;bootm $loadaddr - e00000
Using FM1@DTSEC1 device
TFTP from server 10.116.65.126; our IP address is 10.116.65.101
Filename 'uImage'.
Load address: 0x1000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
####################################
4 MiB/s
done
Bytes transferred = 5175642 (4ef95a hex)
Using FM1@DTSEC1 device
TFTP from server 10.116.65.126; our IP address is 10.116.65.101
Filename 'uImage-t4240rdb.dtb'.
Load address: 0xe00000
Loading: ############
4.1 MiB/s
done
Bytes transferred = 60481 (ec41 hex)
WARNING: adjusting available memory to 30000000
## Booting kernel from Legacy Image at 01000000 ...
Image Name: Linux-4.1.8-rt8+gbd51baf
Created: 2016-11-23 5:04:46 UTC
Image Type: PowerPC Linux Kernel Image (gzip compressed)
Data Size: 5175578 Bytes = 4.9 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 00e00000
Booting using the fdt blob at 0x00e00000
Uncompressing Kernel Image ... OK
Loading Device Tree to 03fde000, end 03fffc40 ... OK
Using fsl_fm_max_frm=9600 from bootargs
Using CoreNet Generic machine description
MMU: Supported page sizes
4 KB as direct
2048 KB as direct & indirect
4096 KB as direct
16384 KB as direct
65536 KB as direct
262144 KB as direct
1048576 KB as direct
MMU: Book3E HW tablewalk enabled
bootconsole [udbg0] enabled
CPU maps initialized for 2 threads per core
Starting Linux PPC64 #4 SMP Wed Nov 23 10:34:28 IST 2016
-----------------------------------------------------
ppc64_pft_size = 0x0
phys_mem_size = 0x180000000
dcache_line_size = 0x40
icache_line_size = 0x40
cpu_features = 0x00180480581802c8
possible = 0x00180480581802c8
always = 0x00180400581802c0
cpu_user_features = 0xdc008000 0x08000000
mmu_features = 0x000a0010
firmware_features = 0x0000000000000000
-----------------------------------------------------
<- setup_system()
Linux version 4.1.8-rt8+gbd51baf (nslab12@nslab12-Veriton-Series) (gcc version 4.9.2 (GCC) ) #4 SMP Wed Nov 23 10:34:28 IST 2016
CoreNet Generic board
Zone ranges:
DMA [mem 0x0000000000000000-0x000000007fffffff]
DMA32 empty
Normal [mem 0x0000000080000000-0x000000017fffffff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000017fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000017fffffff]
MMU: Allocated 2112 bytes of context maps for 255 contexts
PERCPU: Embedded 17 pages/cpu @c00000017a600000 s28824 r0 d40808 u131072
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 1551360
Kernel command line: root=/dev/mmcblk0p1 rw rootfstype=ext2 rootwait rootdelay=2 console=ttyS0,115200 fsl_fm_max_frm=9600
log_buf_len individual max cpu contribution: 4096 bytes
log_buf_len total cpu_extra contributions: 94208 bytes
log_buf_len min size: 16384 bytes
log_buf_len: 131072 bytes
early log buf free: 12376(75%)
PID hash table entries: 4096 (order: 3, 32768 bytes)
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Sorting __ex_table...
Memory: 6108940K/6291456K available (7972K kernel code, 1144K rwdata, 3012K rodata, 360K init, 764K bss, 182516K reserved, 0K cma-reserved)
Hierarchical RCU implementation.
RCU debugfs-based tracing is enabled.
CONFIG_RCU_FANOUT set to non-default value of 32
Additional per-CPU info printed with stalls.
NR_IRQS:512 nr_irqs:512 16
mpic: Setting up MPIC " OpenPIC " version 1.2 at ffe040000, max 24 CPUs
mpic: ISU size: 512, shift: 9, mask: 1ff
mpic: Initializing for 512 sources
clocksource timebase: mask: 0xffffffffffffffff max_cycles: 0xa9210e89c, max_idle_ns: 440795203878 ns
clocksource: timebase mult[15d17460] shift[24] registered
Console: colour dummy device 80x25
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 16384 (order: 5, 131072 bytes)
Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes)
e6500 family performance monitor hardware support registered
Brought up 24 CPUs
devtmpfs: initialized
clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
xor: measuring software checksum speed
8regs : 3354.000 MB/sec
8regs_prefetch: 2979.000 MB/sec
32regs : 2823.000 MB/sec
32regs_prefetch: 2489.000 MB/sec
altivec : 5386.000 MB/sec
xor: using function: altivec (5386.000 MB/sec)
NET: Registered protocol family 16
Bman ver:0a02,02,01
qman-fqd addr 0x0 size 0x400000
------------[ cut here ]------------
kernel BUG at /media/nslab12/work2/Yocto/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb/tmp/work-shared/t4240rdb/kernel-source/drivers/staging/fsl_qbman/qman_config.c:521!
Oops: Exception in kernel mode, sig: 5 [#1]
SMP NR_CPUS=24 CoreNet Generic
Modules linked in:
CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.1.8-rt8+gbd51baf #4
task: c00000017805cb10 ti: c0000001780a0000 task.ti: c0000001780a0000
NIP: c000000000af0ab8 LR: c000000000af0ab0 CTR: c00000000001eb78
REGS: c0000001780a3790 TRAP: 0700 Not tainted (4.1.8-rt8+gbd51baf)
MSR: 0000000080029000 <CE,EE,ME> CR: 24adbe22 XER: 20000000
SOFTE: 1
GPR00: c000000000af0ab0 c0000001780a3a10 c000000000c2c400 000000000000001f
GPR04: 0000000000000001 0000000000000000 0000000000000067 c000000000b8fa30
GPR08: 0000000000000000 fffffffffffffff4 c000000000b1f108 0000000000000006
GPR12: 0000000022adbe22 c00000003fff56c0 c0000000000021cc 0000000000000000
GPR16: 0000000000000000 0000000000000000 c000000000a40e00 c000000000a42d18
GPR20: c000000000ce6e88 c000000000a42c48 c000000000a42c70 c000000000a42c18
GPR24: c000000000a42c98 c000000000a40e70 c000000000a42ce8 c000000000a42cb8
GPR28: c000000000b56d38 c000000000ce6e80 c000000000a01420 c00000017fff9a00
NIP [c000000000af0ab8] .qman_init_early+0x1dc/0x41c
LR [c000000000af0ab0] .qman_init_early+0x1d4/0x41c
Call Trace:
[c0000001780a3a10] [c000000000af0ab0] .qman_init_early+0x1d4/0x41c (unreliable)
[c0000001780a3b70] [c00000000000197c] .do_one_initcall+0xb4/0x22c
[c0000001780a3c60] [c000000000ac1e28] .kernel_init_freeable+0x190/0x2a8
[c0000001780a3db0] [c0000000000021e8] .kernel_init+0x1c/0xe34
[c0000001780a3e30] [c000000000000998] .ret_from_kernel_thread+0x58/0xc0
Instruction dump:
41be0200 7ee3bb78 4bcd0011 60000000 39200000 e89d0008 7ea3ab78 e8bc0000
912100ac 4bccfff5 60000000 e92100ae <0b090000> e89d0010 39200000 e8bc0008
---[ end trace a1a9ca8513bef9d2 ]---

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005

Rebooting in 180 seconds..

[sareenakp]

Thanks Scott for the reply.

Yes the  dtb has the reserved memory section with qman-fqd in it.

I got the uImage-t4240rdb.dtb from SDK 2.0 image ISO. (Attached for reference)

I did not notice the error - a image is not a fdt issue and the fdt from the flash was taken.

I fixed the issue as per U-Boot error after enabling ftrace in kernel uImage: image is not a fdt - must RESET the board to re... 

Now the crash with qman_config.c is solved.

However, the kernel crashes/rather hangs at later point

libphy: Freescale XGMAC MDIO Bus: probed  (Logs attached - SDK2_0_BootLog.txt)

Kindly guide me forward.

Thanks,
Sareena.

[sareenakp]

I was able to solve the hang issue by disabling Freescale XGMAC MDIO  in the menuconfig.

Thanks the issue is solved by providing the right fdt file.

[scottwood · Accepted solution]

The crash is due to a reserved memory region not having been assigned.  Where did your dtb come from?  Does it have a "reserved-memory" node with several child nodes including "qman-fqd"?

[sareenakp]

With disabling the QMan and BMan from menuconfig, we are able to avert the crash. But after the linux 4.18 has come up, no network interfaces  (Eth0-11 ) are seen in ifconfig -a as well as in 'ip link show' command.

I am using the uImage-t4240rdb.dtb file. Can any one please let me know the way forward to avert the crash as well as get all network interface in the kernel? Thanks in advance for the help as this is impacting our deadline.

Thanks,

Sareena.