LS1043A-RGW secondary CPUs fail to come online

[andrewbarrow]

I'm running LS1043A-RDB firmware I built with the LSDK, with hand-edited DTB file. The secondary CPUs fail to come online at boot. This happens with either 4.4 or 4.9 kernel.

I boot by U-Boot loading image and the DTB from SD card. 

NAND contents are the same as the RGW board comes with.

Am I missing a firmware image? Do I have a bad DTB? 

[andrewbarrow]

Also is there a pre-compiled u-boot-sdboot.bin somewhere for the LS1043A-RGW (residential gateway)?

[j_wallace]

Hello Andrew.

Did you solve your issue?

B.R.

[andrewbarrow]

We did - we got a LS1043ARDB board and moved away from the ARGW board. Thanks.

[j_wallace]

Ok. Thanks for your feedback.

Best Regards.

[andrewbarrow]

I have LS1043A RGW (residential gateway) board and it has the newest u-boot image from the ASK loadable binaries, so I can't load the newer RDB u-boot onto it.

I am trying to load a Ubuntu image I made with LSDK and a DTB I hand-edited. Everything is working except I have no secondary CPUs coming up.

[Pavel]

Change u-boot on your LS1043ARDB board. Similar behavior happens if u-boot from previous version is used.

Have a great day,
Pavel Chubakov

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[andrewbarrow]

Here are boot messages from RDB (boot freezes) and RGW (boots, no secondary CPUs)... differences highlighted:

Default LS1043ARDB LSDK file: fsl-ls1043a-rdb.dtb - boot freezes prior to CPUs coming online:

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.9.62 (root@saturn.ap.freescale.net) (gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) ) #1 SMP PREEMPT Sat Dec 23 00:28:49 CST 2017
[ 0.000000] Boot CPU: AArch64 Processor [410fd034]
[ 0.000000] earlycon: uart8250 at MMIO 0x00000000021c0500 (options '')
[ 0.000000] bootconsole [uart8250] enabled
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] OF: reserved mem: initialized node qman-fqd, compatible id fsl,qman-fqd
[ 0.000000] OF: reserved mem: initialized node qman-pfdr, compatible id fsl,qman-pfdr
[ 0.000000] OF: reserved mem: initialized node usdpaa_mem, compatible id fsl,usdpaa-mem
[ 0.000000] OF: reserved mem: initialized node bman-fbpr, compatible id fsl,bman-fbpr
[ 0.000000] cma: Reserved 16 MiB at 0x00000000fb000000
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000000ffdfffff]
[ 0.000000] NUMA: Adding memblock [0x80000000 - 0xfbffffff] on node 0
[ 0.000000] NUMA: Adding memblock [0xff000000 - 0xff3fffff] on node 0
[ 0.000000] NUMA: Adding memblock [0xff800000 - 0xffdfffff] on node 0
[ 0.000000] NUMA: Initmem setup node 0 [mem 0x80000000-0xffdfffff]
[ 0.000000] NUMA: NODE_DATA [mem 0xffdc74c0-0xffdc8c7f]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000080000000-0x00000000ffdfffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000080000000-0x00000000fbffffff]
[ 0.000000] node 0: [mem 0x00000000ff000000-0x00000000ff3fffff]
[ 0.000000] node 0: [mem 0x00000000ff800000-0x00000000ffdfffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffdfffff]
[ 0.000000] WARNING: x1-x3 nonzero in violation of boot protocol:
[ 0.000000] x1: 0000000000000000
[ 0.000000] x2: 0000000000000000
[ 0.000000] x3: 0000000080080000
[ 0.000000] This indicates a broken bootloader or old kernel
[ 0.000000] percpu: Embedded 23 pages/cpu @ffff80007fd66000 s55000 r8192 d31016 u94208
[ 0.000000] Detected VIPT I-cache on CPU0
[ 0.000000] CPU features: enabling workaround for ARM erratum 845719
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 502280
[ 0.000000] Policy zone: DMA
[ 0.000000] Kernel command line: root=/dev/mmcblk0p3 rw rootdelay=5 console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Memory: 1709296K/2041856K available (12412K kernel code, 1054K rwdata, 4896K rodata, 1152K init, 978K bss, 316176K reserved, 16384K cma-reserved)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] modules : 0xffff000000000000 - 0xffff000008000000 ( 128 MB)
[ 0.000000] vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000 (129022 GB)
[ 0.000000] .text : 0xffff000008080000 - 0xffff000008ca0000 ( 12416 KB)
[ 0.000000] .rodata : 0xffff000008ca0000 - 0xffff000009170000 ( 4928 KB)
[ 0.000000] .init : 0xffff000009170000 - 0xffff000009290000 ( 1152 KB)
[ 0.000000] .data : 0xffff000009290000 - 0xffff000009397a00 ( 1055 KB)
[ 0.000000] .bss : 0xffff000009397a00 - 0xffff00000948c5d4 ( 979 KB)
[ 0.000000] fixed : 0xffff7dfffe7fd000 - 0xffff7dfffec00000 ( 4108 KB)
[ 0.000000] PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000 ( 16 MB)
[ 0.000000] vmemmap : 0xffff7e0000000000 - 0xffff800000000000 ( 2048 GB maximum)
[ 0.000000] 0xffff7e0000000000 - 0xffff7e0001ff8000 ( 31 MB actual)
[ 0.000000] memory : 0xffff800000000000 - 0xffff80007fe00000 ( 2046 MB)
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 64.
[ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4
[ 0.000000] NR_IRQS:64 nr_irqs:64 0
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] GIC CPU mask not found - kernel will fail to boot.
[ 0.000000] GIC CPU mask not found - kernel will fail to boot.
[ 0.000000] arm_arch_timer: Enabling workaround for FSL erratum A-008585
[ 0.000000] arm_arch_timer: Architected cp15 timer(s) running at 25.00MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[ 0.000002] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[ 0.008460] Console: colour dummy device 80x25
[ 0.012951] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[ 0.023350] pid_max: default: 32768 minimum: 301
[ 0.028054] Security Framework initialized
[ 0.032460] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[ 0.040756] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.048338] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.055071] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.062789] ASID allocator initialised with 65536 entries

--- boot freezes here ---

***

My .dtb, based on ls1043a-rgw DTB image for OpenWRT pre-compiled binary (boots, but secondary CPUs do not come online):

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.9.62 (root@saturn.ap.freescale.net) (gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) ) #1 SMP PREEMPT Sat Dec 23 00:28:49 CST 2017
[ 0.000000] Boot CPU: AArch64 Processor [410fd034]
[ 0.000000] earlycon: uart8250 at MMIO 0x00000000021c0500 (options '')
[ 0.000000] bootconsole [uart8250] enabled
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] OF: reserved mem: initialized node qman-fqd, compatible id fsl,qman-fqd
[ 0.000000] OF: reserved mem: initialized node qman-pfdr, compatible id fsl,qman-pfdr
[ 0.000000] OF: reserved mem: initialized node bman-fbpr, compatible id fsl,bman-fbpr
[ 0.000000] cma: Reserved 16 MiB at 0x00000000fb000000
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000000ffdfffff]
[ 0.000000] NUMA: Adding memblock [0x80000000 - 0xfbffffff] on node 0
[ 0.000000] NUMA: Adding memblock [0xff800000 - 0xffdfffff] on node 0
[ 0.000000] NUMA: Initmem setup node 0 [mem 0x80000000-0xffdfffff]
[ 0.000000] NUMA: NODE_DATA [mem 0xffdc8880-0xffdca03f]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000080000000-0x00000000ffdfffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000080000000-0x00000000fbffffff]
[ 0.000000] node 0: [mem 0x00000000ff800000-0x00000000ffdfffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffdfffff]
[ 0.000000] WARNING: x1-x3 nonzero in violation of boot protocol:
[ 0.000000] x1: 0000000000000000
[ 0.000000] x2: 0000000000000000
[ 0.000000] x3: 0000000080080000
[ 0.000000] This indicates a broken bootloader or old kernel
[ 0.000000] percpu: Embedded 23 pages/cpu @ffff80007fd67000 s55000 r8192 d31016 u94208
[ 0.000000] Detected VIPT I-cache on CPU0
[ 0.000000] CPU features: enabling workaround for ARM erratum 845719
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 501256
[ 0.000000] Policy zone: DMA
[ 0.000000] Kernel command line: root=/dev/mmcblk0p3 rw rootdelay=5 console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Memory: 1967348K/2037760K available (12412K kernel code, 1054K rwdata, 4896K rodata, 1152K init, 978K bss, 54028K reserved, 16384K cma-reserved)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] modules : 0xffff000000000000 - 0xffff000008000000 ( 128 MB)
[ 0.000000] vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000 (129022 GB)
[ 0.000000] .text : 0xffff000008080000 - 0xffff000008ca0000 ( 12416 KB)
[ 0.000000] .rodata : 0xffff000008ca0000 - 0xffff000009170000 ( 4928 KB)
[ 0.000000] .init : 0xffff000009170000 - 0xffff000009290000 ( 1152 KB)
[ 0.000000] .data : 0xffff000009290000 - 0xffff000009397a00 ( 1055 KB)
[ 0.000000] .bss : 0xffff000009397a00 - 0xffff00000948c5d4 ( 979 KB)
[ 0.000000] fixed : 0xffff7dfffe7fd000 - 0xffff7dfffec00000 ( 4108 KB)
[ 0.000000] PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000 ( 16 MB)
[ 0.000000] vmemmap : 0xffff7e0000000000 - 0xffff800000000000 ( 2048 GB maximum)
[ 0.000000] 0xffff7e0000000000 - 0xffff7e0001ff8000 ( 31 MB actual)
[ 0.000000] memory : 0xffff800000000000 - 0xffff80007fe00000 ( 2046 MB)
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 64.
[ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4
[ 0.000000] NR_IRQS:64 nr_irqs:64 0
[ 0.000000] GIC: Adjusting CPU interface base to 0x000000000142f000
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] arm_arch_timer: Enabling workaround for FSL erratum A-008585
[ 0.000000] arm_arch_timer: Architected cp15 timer(s) running at 25.00MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[ 0.000002] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[ 0.008483] Console: colour dummy device 80x25
[ 0.012980] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[ 0.023385] pid_max: default: 32768 minimum: 301
[ 0.028091] Security Framework initialized
[ 0.032501] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[ 0.040802] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.048388] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.055126] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.062838] ASID allocator initialised with 65536 entries
[ 0.100790] EFI services will not be available.
[ 1.149902] CPU1: failed to come online
[ 1.153749] CPU1: failed in unknown state : 0x0
[ 2.210854] CPU2: failed to come online
[ 2.214701] CPU2: failed in unknown state : 0x0
[ 3.271804] CPU3: failed to come online
[ 3.275650] CPU3: failed in unknown state : 0x0
[ 3.280227] Brought up 1 CPUs

[andrewbarrow]

Here are some notable differences in the .dtb files that boot on one cpu only vs. fail to boot:

Default LS1043ARDB LSDK file: fsl-ls1043a-rdb.dtb - boot freezes prior to CPUs coming online:

interrupt-controller@1400000 {
   compatible = "arm,gic-400";
   #interrupt-cells = <0x3>;
   interrupt-controller;
   reg = <0x0 0x1401000 0x0 0x1000 0x0 0x1402000 0x0 0x2000 0x0 0x1404000 0x0 0x2000 0x0 0x1406000 0x0       0x2000>;
   interrupts = <0x1 0x9 0xf08>;
   linux,phandle = <0x1>;
   phandle = <0x1>;
};

firmware {

   optee {
   compatible = "linaro,optee-tz";
   method = "smc";
};

My .dtb, based on ls1043a-rgw DTB image for OpenWRT pre-compiled binary (boots, but secondary CPUs do not come online):

interrupt-controller@1400000 {
   compatible = "arm,gic-400";
   #interrupt-cells = <0x3>;
   interrupt-controller;
   reg = <0x0 0x1410000 0x0 0x10000 0x0 0x1420000 0x0 0x20000 0x0 0x1440000 0x0 0x20000 0x0 0x1460000 0x0    0x20000>;
   interrupts = <0x1 0x9 0xf08>;
   linux,phandle = <0x1>;
   phandle = <0x1>;
};

- note my dtb that boots on primary only has no OPTEE firmware 

So I'm thinking one of the following could be the problem:

1. the RGW board is missing the OPTEE firmware or PPA firmware (I haven't loaded any because I found no instructions for loading it to NAND on the RGW board). With no secure boot firmware, only CPU (needed for boot) comes up. 

2. the reg values for interrupt-controller are incorrect in the first case, giving "GIC CPU mask not found - kernel will fail to boot" error

3. I couldn't get booting from SD card to work on my RGW board, so I am booting from NAND with the following u-boot:

sdboot=setenv bootargs root=/dev/mmcblk0p3 rw rootdelay=5 console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500;mmcinfo;ext2load mmc 0:2 $loadaddr Image; ext2load mmc 0:2 a0000000 fsl-ls1043a-rdb.dtb; booti $loadaddr - a0000000

If the MMC image has e.g. OPTEE or PPA firmware present, I must be ignoring it as I only fetch kernel and dtb from MMC.

[Pavel]

It looks like that your DTB file is incorrect. Test your board using default version of the LS1043ARDB LSDK DTB file.

Have a great day,
Pavel Chubakov

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------