LS1046A Kernel Panic (DDR Calibration Problem?)

[hyeonbin]

Hello, I am currently designing and testing based on LS1046FRWY board, but Kernel Panic has occurred, so I am inquiring.

The symptoms are not the initial sensors command, but when the cpu temperature is about 50 degrees Celsius, the following message appears when the reboot command is issued.

The system is currently under development with Fanless, and the problem below disappears when Fan is turned on, so I hope it becomes a reboot command even if the temperature rises (70 degrees Celsius or higher).

We built another system based on LS1046a, and it was fine up to 75 degrees Celsius. But there's an unexpected problem with the newly developed system.

So I suspect DDR Timing problem and we have NXP Code Warrior TAP product. I hope we can solve it through this.

NOTICE: Fixed DDR on board

NOTICE: 4 GB DDR4, 64-bit, CL=15, ECC on
NOTICE: BL2: v1.5(release):LSDK-20.12-dirty
NOTICE: BL2: Built : 10:19:35, Apr 7 2022
NOTICE: BL31: v1.5(release):LSDK-20.12-dirty
NOTICE: BL31: Built : 10:19:35, Apr 7 2022
NOTICE: Welcome to LS1046 BL31 Phase

U-Boot 2020.04-dirty (Apr 07 2022 - 10:19:10 +0900)

SoC: LS1046AE Rev1.0 (0x87070010)
Clock Configuration:
CPU0(A72):1200 MHz CPU1(A72):1200 MHz CPU2(A72):1200 MHz
CPU3(A72):1200 MHz
Bus: 500 MHz DDR: 2100 MT/s FMAN: 700 MHz
Reset Configuration Word (RCW):
00000000: 0a15000c 0e000000 00000000 00000000
00000010: 11338888 00400016 60040000 c1000000
00000020: 00000000 00000000 00000000 00038800
00000030: 20044100 24003101 00000096 00000001
Model: LS1046A FRWY Board
Board: LS1046AFRWY-SATURN-HURA, Rev: B, boot from SD
SD1_CLK1 = 100.00MHZ, SD1_CLK2 = 100.00MHZ
DRAM: 3.9 GiB (DDR4, 64-bit, CL=15, ECC on)
SEC0: RNG instantiated
select_i2c_ch_pca9547: Cannot find udev for a bus 0
Using SERDES1 Protocol: 4403 (0x1133)
Using SERDES2 Protocol: 34952 (0x8888)
NAND: 0 MiB
MMC: FSL_SDHC: 0
Loading Environment from MMC... OK
EEPROM: Read failed.
In: serial
Out: serial
Err: serial
Net:
MMC read: dev # 0, block # 18432, count 128 ...
Fman1: Uploading microcode version 106.4.18
Could not get PHY for FSL_MDIO0: addr 1
Failed to connect
PCIe1: pcie@3400000 Root Complex: no link
PCIe2: pcie@3500000 disabled
PCIe3: pcie@3600000 disabled
FM1@DTSEC5, FM1@DTSEC6, FM1@TGEC1, FM1@TGEC2
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Scanning mmc 0:2...
Found U-Boot script /ls1046afrwy_boot.scr
15431132 bytes read in 1017 ms (14.5 MiB/s)
do_fpgaload: fpga_load 15431132 bytes in 2721 ms
968 bytes read in 16 ms (58.6 KiB/s)
## Executing script at 80000000
37083648 bytes read in 2430 ms (14.6 MiB/s)
31419 bytes read in 17 ms (1.8 MiB/s)
## Flattened Device Tree blob at 90000000
Booting using the fdt blob at 0x90000000
Loading Device Tree to 000000008ffe5000, end 000000008ffffaba ... OK
WARNING failed to get smmu node: FDT_ERR_NOTFOUND
WARNING failed to get smmu node: FDT_ERR_NOTFOUND

Starting kernel ...

[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd082]
[ 0.000000] Linux version 5.4.47-dirty (root@hura-H81M-D2V) (gcc version 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)) #1 SMP PREEMPT Thu Apr 7 10:03:32 KST 2022
[ 0.000000] Machine model: LS1046A FRWY Board
[ 0.000000] earlycon: uart8250 at MMIO 0x00000000021c0500 (options '')
[ 0.000000] printk: bootconsole [uart8250] enabled
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] OF: reserved mem: initialized node qman-fqd, compatible id fsl,qman-fqd
[ 0.000000] OF: reserved mem: initialized node qman-pfdr, compatible id fsl,qman-pfdr
[ 0.000000] OF: reserved mem: initialized node bman-fbpr, compatible id fsl,bman-fbpr
[ 0.000000] cma: Reserved 320 MiB at 0x00000000e7c00000
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x00000008ff7fffff]
[ 0.000000] NUMA: NODE_DATA [mem 0x8ff010800-0x8ff011fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA32 [mem 0x0000000080000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x00000008ff7fffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000080000000-0x00000000fbdfffff]
[ 0.000000] node 0: [mem 0x0000000880000000-0x00000008fbffffff]
[ 0.000000] node 0: [mem 0x00000008ff000000-0x00000008ff7fffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000008ff7fffff]
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
[ 0.000000] psci: SMC Calling Convention v1.1
[ 0.000000] percpu: Embedded 24 pages/cpu s59928 r8192 d30184 u98304
[ 0.000000] Detected PIPT I-cache on CPU0
[ 0.000000] CPU features: detected: EL2 vector hardening
[ 0.000000] CPU features: kernel page table isolation forced ON by KASLR
[ 0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[ 0.000000] Speculative Store Bypass Disable mitigation not required
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1001184
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500 root=PARTUUID=4052baa9-04 rw rootwait
[ 0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] ------------[ cut here ]------------
[ 0.000000] kernel BUG at /home/hura1/flexbuild_lsdk2012_ls1046_D1_2/packages/linux/linux/arch/arm64/kernel/acpi.c:284!
[ 0.000000] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.4.47-dirty #1
[ 0.000000] Hardware name: LS1046A FRWY Board (DT)
[ 0.000000] pstate: 80000185 (Nzcv dAIf -PAN -UAO)
[ 0.000000] pc : apei_claim_sea+0xc8/0x100
[ 0.000000] lr : apei_claim_sea+0x28/0x100
[ 0.000000] sp : ffffbfbb8a5736c0
[ 0.000000] x29: ffffbfbb8a5736c0 x28: ffffbfbb8a584b40
[ 0.000000] x27: 0000000000000000 x26: ffffbfbb8a09a008
[ 0.000000] x25: ffff00087f012200 x24: 0000000000000021
[ 0.000000] x23: 0000000060000185 x22: ffffbfbb884a0bb8
[ 0.000000] x21: ffffbfbb8a573750 x20: 0000000000000180
[ 0.000000] x19: ffffbfbb8a584b40 x18: ffffffffffffffff
[ 0.000000] x17: 0000000000018000 x16: 0000000000000000
[ 0.000000] x15: ffffbfbb8a579908 x14: 00000008ff03c000
[ 0.000000] x13: 0000000400000000 x12: 00000000000298bc
[ 0.000000] x11: 00000008ff012740 x10: 0000000400000000
[ 0.000000] x9 : 0000000000000000 x8 : ffff000066288000
[ 0.000000] x7 : 0000000000000000 x6 : 000000000000003f
[ 0.000000] x5 : 0000000080000000 x4 : 0000000000000000
[ 0.000000] x3 : 0000000080000000 x2 : ffff404cede74000
[ 0.000000] x1 : ffffbfbb8a584b40 x0 : 0000000000110002
[ 0.000000] Call trace:
[ 0.000000] apei_claim_sea+0xc8/0x100
[ 0.000000] do_sea+0x38/0x68
[ 0.000000] do_mem_abort+0x3c/0x98
[ 0.000000] el1_da+0x1c/0x90
[ 0.000000] do_sea+0x0/0x68
[ 0.000000] el1_da+0x1c/0x90
[ 0.000000] __rcu_read_unlock+0x0/0x190
[ 0.000000] __stack_chk_guard+0x0/0x8
[ 0.000000] Code: d5384113 b9401260 f90013b5 36a7fba0 (d4210000)
[ 0.000000] random: get_random_bytes called from print_oops_end_marker+0x48/0x68 with crng_init=0
[ 0.000000] ---[ end trace 0000000000000000 ]---
[ 0.000000] Kernel panic - not syncing: Fatal exception in interrupt

[yipingwang]

Please use QCVS DDRv tool to connect to your target board to do DDR validation and optimization.

Please download and install CodeWarrior from the following link.

https://drive.google.com/file/d/1Kjq1nLYrtIfWAHYrup5Cf-TTEJI9iWiu/view?usp=sharing

Please download com.freescale.armv8.11.5.5.GA.Win.updatesite.210930.zip from the following link, open CodeWarrior IDE and install this service pack from Help->Install New Software->Add->Archive.

https://drive.google.com/file/d/1ZHiYWDOhb27o0oD5Brqjn4_vJzRUCKoY/view?usp=sharing

Then create a QCVS DDR project to connect to the target board to do optimization. Please refer to the attached user guide. If there is SPD on your target board, please create a project with "read from SPD" method.

After get the optimized DDR configuration parameters, please use them to edit ATF source file flexbuild_lsdk2012/packages/firmware/atf/plat/nxp/soc-ls1046/ls1046afrwy/ddr_init.c.

[hyeonbin]

Hello.

I tried DDR Validation after listening to you. But here are the following symptoms.

1. Using LSDK, the Linux login screen appears well.
2. The Codewarrier was connected to load the properties and tested through Validation, but some passed and some failed.

- Pass : Stress test (Data is Address / Row hop read / SSM memcpy x32 / byte SSN memcpy x32 / Memcpy pseudo random data pattern / IRAM to ddr x32 / IRAM to ddr v2 x32)
MemTester(Stuck Address / Random Value / Compare XOR / SUB / Mul / Div / Or / And / Sequential Increment / Solid Bit / Block Sequential / Checkerboard / Bit Spread / Bit Flip / Walkbits1 , 0),
Read Margin, Read ODT and driver, Write ODT and driver, Operational DDR Test

- Fail : Centering the clock 0.12% Fail (WRLVL_START Searcher investigate HW issues on the board. / DDR interfaces is failing due to an issue other than WRLVL_START Values), Write Margin Fail 5.2% Fail(Change the test execution timeout parameter)

In the 'Centering the clock' item, Auto search & detect for write leveling start values, the value of the Search tab appears to be 0/1 and stops.

I also used an app called Stress to load it
The Kernel Panic message appears at the kernel end when testing is performed through the 'stress --vm2 --vm-bytes 128M --timeout 2s' command, but PASS appears using the 'stress --vm2 --vm-bytes 1M --timeout 2s' command.

So I think it's a hardware or artwork problem, but there seems to be no circuit problem. Can we solve this problem?