Help porting custom LS1046A board to u-boot. MMU crash.

[AbelianMeme]

Hi Everyone,

I am attempting to bring our custom PC with an LS1046a up under U-boot. I am using the most current version of U-boot available from github, as we need some features from it.

The issue I have: The instant I try and enable the MMU after setting up the page tables, (in final_mmu_setup() in fsl-layerscape/cpu.c) I immediately generate a synchronous exception upon issuing the set_sctlr() with the CR_M bit. With the exception of the device tree which I necessarily had to modify, I have basically copied the settings directly from ls1046ardb, so I'm not sure what could be going wrong.

Does anyone know what could be causing u-boot to generate a synchronous exception the instant the final MMU is activated?

I printed out the final_map table I am using just before the error. I also added a mapping from 0xffffffff00000000 (idx #7) because I thought that might be the issue (the elr is set to a funny value in the Synchronous Abort Exception), but it didn't help.

U-Boot 2024.04-00900-gc086852891-dirty (Apr 24 2024 - 09:43:08 +0700)

SoC:  LS1046AE Rev1.0 (0x87070010)
Clock Configuration:
       CPU0(A72):1800 MHz  CPU1(A72):1800 MHz  CPU2(A72):1800 MHz  
       CPU3(A72):1800 MHz  
       Bus:      600  MHz  DDR:      2100 MT/s  FMAN:     700  MHz
Reset Configuration Word (RCW):
       00000000: 0c150012 0e000000 00000000 00000000
       00000010: 00008888 00c0001e 40000000 c1000000
       00000020: 00000000 00000000 00000000 00018ffc
       00000030: 20244500 25603100 00000096 00000001
Model: LS1046A Copier Board
Board: Nexcopy LS1046A, boot from QSPI
DRAM:  Initializing DDR....using SPD
Detected UDIMM JM2666HLH-4G      
       DRAM setup success
4 GiB (effective 2 GiB) (DDR4, 64-bit, CL=15, ECC off)
ec1
final_mmu_setup running at EL3, SP=0x00000000ffe12bc0
VM map:  0| 0x         0 0x         0 0x   1000000,
VM map:  1| 0x   1000000 0x   1000000 0x   f000000,
VM map:  2| 0x  10000000 0x  10000000 0x    200000,
VM map:  3| 0x  20000000 0x  20000000 0x   4000000,
VM map:  4| 0x  40000000 0x  40000000 0x  20000000,
VM map:  5| 0x  60000000 0x  60000000 0x  20000000,
VM map:  6| 0x  80000000 0x  80000000 0x  80000000,
VM map:  7| 0xffffffff80000000 0x  80000000 0x  80000000,
VM map:  8| 0x 500000000 0x 500000000 0x  10000000,
VM map:  9| 0x 880000000 0x 880000000 0x         0,
VM map: 10| 0x4000000000 0x4000000000 0x 800000000,
VM map: 11| 0x4800000000 0x4800000000 0x 800000000,
VM map: 12| 0x5000000000 0x5000000000 0x 800000000,

Synchronous Abort Exception, esr 0x8600000e , far[3] 0x4014a350
elr: ffffffff80331350 lr : 0000000040149b40 (reloc)
elr: 000000004014a350 lr : 00000000fff62b40
x0 : 000000001000e508 x1 : 0000000000000042
x2 : 000000004014a350 x3 : 0000000000000001
x4 : 0000000000000028 x5 : 0000000080820000
x6 : 0000000000000003 x7 : 0000000000000000
x8 : 00000000fffc3a50 x9 : 000000000000000c
x10: 000000000a200023 x11: 0000000000000002
x12: 0000000000000002 x13: 00000008c0000000
x14: 000000000000000a x15: 00000000fff1aee0
x16: 0000000000000000 x17: 0000000000000064
x18: 00000000ffe14da0 x19: 000000001000e508
x20: 0000000000000042 x21: 0000000040189320
x22: 00000000fffb67b3 x23: 00000000ffff0000
x24: 0000000880000000 x25: 0000008800000000
x26: 0000000080000000 x27: 0000000000000000
x28: 00000000401039e8 x29: 00000000ffe12b70
SP: 00000000ffe12100 IDR: 0000000080000000

Code: 000000004014a340
   c1 ff ff 97 1f 00 1b 72 e0 17 9f 1a fa ff ff 17 (fd 7b be a9)

The code displayed disassembles to:

0x000000004014a340:  C1 FF FF 97    bl   #0x4014a244
0x000000004014a344:  1F 00 1B 72    tst  w0, #0x20
0x000000004014a348:  E0 17 9F 1A    cset w0, eq
0x000000004014a34c:  FA FF FF 17    b    #0x4014a334
0x000000004014a350:  FD 7B BE A9    stp  x29, x30, [sp, #-0x20]!

The address displayed, 0x4014a350, is the entry point to the function ns16550_serial_putc() [drivers/serial/ns16550.c]

Every crash always occurs on this same instruction, even if I move code around. But there is nothing wrong that I can see with SP as reported in the exception.

We are running at EL3 right now. No trusted architecture or secure memory is configured.

Has anyone ever ported u-boot before, and have any idea what might be happening here?

[yipingwang]

1. For DDR porting, please check whether you have modified the following section in board/freescale/ls1046ardb/ddr.h.

static const struct board_specific_parameters udimm0[] = {
/*
* memory controller 0
* num| hi| rank| clk| wrlvl | wrlvl | wrlvl
* ranks| mhz| GB |adjst| start | ctl2 | ctl3
*/
{2, 1350, 0, 8, 6, 0x0708090B, 0x0C0D0E09,},
{2, 1666, 0, 8, 7, 0x08090A0C, 0x0D0F100B,},
{2, 1900, 0, 8, 7, 0x09090B0D, 0x0E10120B,},
{2, 2300, 0, 8, 7, 0x08090A0E, 0x1011120C,},
{}
};

2. You could debug u-boot with CodeWarrior for ARMv8.

https://community.nxp.com/t5/CodeWarrior-for-QorIQ-Knowledge/Use-CodeWarrior-for-ARMv8-to-Debug-U-bo...

3. Please try u-boot release from the latest NXP LSDK.

https://github.com/nxp-qoriq/u-boot.git

tag: lf-6.1.55-2.2.0

4. In NXP recent released LSDK, we use atf+u-boot, not only use u-boot directly, you could try to use atf+u-boot.

[AbelianMeme]

I have determined the cause of the Synchronous Exception. The serial port driver is still active for some reason that I can't understand. final_mmu_setup() marks the QSPI memory range non executable (UXN and PXN are set). The serial port driver still running out of that area then causes an exception the instant it attempts to run.

I can't figure out what is calling the serial port (is it running on another core, or from an interrupt handler), or how to stop the driver so it can be moved to the new area in RAM. Still looking for help to understand this if anyone has familiarity porting boards to u-boot.