Kernel crash dump | kexec | kdump | initrd load issue issue

kevi49 — Sat, 05 Nov 2022 11:07:50 GMT

Hi NXP Team,

I am working on iMX6sx processor to enable kernel crash dump and system have have 512Mb of RAM.

Followed the "./Documentation/kdump/kdump.txt" and enabled the required driver and configurations.

Enabled driver:

CONFIG_KEXEC=y

CONFIG_CRASH_DUMP=y

CONFIG_PROC_VMCORE=y

CONFIG_SYSFS=y

CONFIG_RELOCATABLE=y

CONFIG_AUTO_ZRELADDR=y

Also able to reserve the crashkernel memory of 64Mb.

console=ttymxc0,115200 loglevel=7 crashkernel=64M retain_initrd root=/dev/ram0 rw uart_from_osc=1 sigtype=dev accept_dev_sigs=y initrd=0x83000000,0x00A43951

And in primary kernel also getting the log of memory reserved for crashkernel

Reserving 64MB of memory at 2304MB for crashkernel (System RAM: 511MB)

Now, I am trying to load the secondary kernel using the kexec -p mode with below command line options.

kexec --type uImage -d -p /uImage --dtb=/imx6sx_35som_better_emmc.dtb --command-line="console=ttymxc0,115200 root=/dev/ram0 maxcpus=1 reset_devices initrd=0x83000000,0x00A43951"

MEMORY RANGES
0000000080000000-000000009fefffff (0)
/przImage header: 0x016f2818 0x00000000 0x003d31a8
zImage size 0x3d31a8, file size 0x3d31a8
zImage requires 0x003e41a8 bytes
Reserved memory ranges
0000000090000000-0000000093ffffff (0)
Coredump memory ranges
0000000080000000-000000008fffffff (0)
0000000094000000-000000009fefffff (0)
kernel symbol _stext vaddr = bd310442
phys offset = 0x80000000, page offset = bc000000
Using 32-bit ELF core format
get_crash_notes_per_cpu: crash_notes addr = 9fa91e00, size = 180
Elf header: p_type = 4, p_offset = 0x9fa91e00 p_paddr = 0x9fa91e00 p_vaddr = 0x0 p_filesz = 0xb4 p_memsz = 0xb4
vmcoreinfo header: p_type = 4, p_offset = 0x8e1a4000 p_paddr = 0x8e1a4000 p_vaddr = 0x0 p_filesz = 0x1024 p_memsz = 0x1024
Elf header: p_type = 1, p_offset = 0x80000000 p_paddr = 0x80000000 p_vaddr = 0xbc000000 p_filesz = 0x10000000 p_memsz = 0x10000000
Elf header: p_type = 1, p_offset = 0x94000000 p_paddr = 0x94000000 p_vaddr = 0xd0000000 p_filesz = 0xbf00000 p_memsz = 0xbf00000
elfcorehdr: 0x93f00000
crashkernel: [0x90000000 - 0x93ffffff] (64M)
memory range: [0x80000000 - 0x8fffffff] (256M)
memory range: [0x94000000 - 0x9fefffff] (191M)
kernel command line: "cachepolicy=writealloc console=ttymxc0,115200 loglevel=7 disable_wd=n root=/dev/ram0 rw uart_from_osc=1 sigtype=dev accept_dev_sigs=y maxcpus=1 r
eset_devices initrd=0x83000000,0x00A43951 elfcorehdr=0x93f00000 mem=64512K"
Kernel: address=0x90008000 size=0x01374848
DT : address=0x9137e000 size=0x0000bd8f
kexec_load: entry = 0x90008000 flags = 0x280001
nr_segments = 3
segment[0].buf = 0x76a99048
segment[0].bufsz = 0x3d31ac
segment[0].mem = 0x90008000
segment[0].memsz = 0x3d4000
segment[1].buf = 0xcd6150
segment[1].bufsz = 0xbd8f
segment[1].mem = 0x9137e000
segment[1].memsz = 0xc000
segment[2].buf = 0xcd1678
segment[2].bufsz = 0x400
segment[2].mem = 0x93f00000
segment[2].memsz = 0x1000

echo c > /proc/sysrq-trigger

after entering above command secondary kernel is started booting with below logs but stuck in loading initrd image.

[ 153.055061] [<80214bf8>] (vfs_write) from [<80214e0c>] (SyS_write+0x4c/0xc0)
[ 153.062123] [<80214e0c>] (SyS_write) from [<80107820>] (__sys_trace_return+0x0/0x20)
[ 153.069873] Code: e5834000 f57ff04e ebeff373 e3a03000 (e5c34000)
[ 153.076130] Loading crashdump kernel...
[ 153.080059] Bye!
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.14.242-00018-g80effdaa9208-dirty (VirtualBox) (gcc version 9.3.0 (GCC)) #71 SMP PREEMPT Sat Nov 5 15:35:09 IST 2022
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c53c7d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: Freescale i.MX6 SoloX SG base Board
[ 0.000000] OF: fdt: search "chosen", depth: 0, uname:
[ 0.000000] OF: fdt: search "chosen", depth: 1, uname: chosen
[ 0.000000] OF: fdt: Looking for initrd properties...
[ 0.000000] OF: fdt: Command line is: console=ttymxc0,115200 root=/dev/ram0 rw maxcpus=1 reset_devices initrd=0x83000000,0x00A43951 elfcorehdr=0x93f00000 mem=64512K
[ 0.000000] OF: fdt: dt_root_size_cells = 1
[ 0.000000] OF: fdt: dt_root_addr_cells = 1
[ 0.000000] OF: fdt: memory scan node memory, reg size 8,
[ 0.000000] OF: fdt: - 80000000 , 1ff00000
[ 0.000000] memblock_add: [0x90000000-0x9fefffff] early_init_dt_scan_memory+0x134/0x188
[ 0.000000] memblock_add: [0x90000000-0x93efffff] arm_add_memory+0x14c/0x174
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] memblock_reserve: [0x90100000-0x909b014f] arm_memblock_init+0x30/0x250
[ 0.000000] ##### arm_initrd_init 243
[ 0.000000] ##### arm_initrd_init 250
[ 0.000000] ##### arm_initrd_init 253 initrd_start: 0x0
[ 0.000000] ##### arm_initrd_init 254 initrd_end: 0x0
[ 0.000000] INITRD: 0x83000000+0x00a44000 is not a memory region - disabling initrd
[ 0.000000] memblock_reserve: [0x90004000-0x90007fff] arm_memblock_init+0x1f4/0x250
[ 0.000000] memblock_reserve: [0x9137e000-0x91389d8e] arm_memblock_init+0x208/0x250

Now secondary kernel booted successfully but initrd image is in the "0x83000000+0x00a44000" memory range and secondary kernel is running on "0x90000000-0x9fefffff" which is not accessible and because of this reason board is getting stuck with below logs.

[ 5.071653] List of all partitions:
[ 5.075163] 0100 131072 ram0
[ 5.075168] (driver?)
[ 5.081451] 0101 131072 ram1
[ 5.081457] (driver?)
[ 5.087573] 0102 131072 ram2
[ 5.087577] (driver?)
[ 5.093881] 0103 131072 ram3
[ 5.093887] (driver?)
[ 5.100001] 0104 131072 ram4
[ 5.100005] (driver?)
[ 5.106178] 0105 131072 ram5
[ 5.106183] (driver?)
[ 5.112323] 0106 131072 ram6
[ 5.112327] (driver?)
[ 5.118440] 0107 131072 ram7
[ 5.118444] (driver?)
[ 5.124614] 0108 131072 ram8
[ 5.124619] (driver?)
[ 5.130732] 0109 131072 ram9
[ 5.130735] (driver?)
[ 5.136889] 010a 131072 ram10
[ 5.136895] (driver?)
[ 5.143115] 010b 131072 ram11
[ 5.143120] (driver?)
[ 5.149318] 010c 131072 ram12
[ 5.149321] (driver?)
[ 5.155566] 010d 131072 ram13
[ 5.155571] (driver?)
[ 5.161791] 010e 131072 ram14
[ 5.161795] (driver?)
[ 5.167994] 010f 131072 ram15
[ 5.167998] (driver?)
[ 5.174227] b300 3760128 mmcblk3
[ 5.174232] driver: mmcblk
[ 5.181039] b301 196608 mmcblk3p1 031b5d3c-01
[ 5.181044]
[ 5.187897] b302 512000 mmcblk3p2 031b5d3c-02
[ 5.187903]
[ 5.194733] b303 512000 mmcblk3p3 031b5d3c-03
[ 5.194737]
[ 5.201566] b304 1 mmcblk3p4
[ 5.201570]
[ 5.207423] b305 65536 mmcblk3p5 031b5d3c-05
[ 5.207426]
[ 5.214277] b306 65536 mmcblk3p6 031b5d3c-06
[ 5.214282]
[ 5.221089] b307 51200 mmcblk3p7 031b5d3c-07
[ 5.221093]
[ 5.227927] 103:00000 51200 mmcblk3p8 031b5d3c-08
[ 5.227930]
[ 5.234782] 103:00001 10240 mmcblk3p9 031b5d3c-09
[ 5.234786]
[ 5.241616] 103:00002 512000 mmcblk3p10 031b5d3c-0a
[ 5.241620]
[ 5.248515] 103:00003 531307 mmcblk3p11 031b5d3c-0b
[ 5.248519]
[ 5.255459] 103:00004 20480 mmcblk3p12 031b5d3c-0c
[ 5.255464]
[ 5.262383] 103:00005 20480 mmcblk3p13 031b5d3c-0d
[ 5.262387]
[ 5.269281] 103:00006 512000 mmcblk3p14 031b5d3c-0e
[ 5.269284]
[ 5.276222] 103:00007 696236 mmcblk3p15 031b5d3c-0f
[ 5.276227]
[ 5.283154] b310 16384 mmcblk3boot1
[ 5.283158] (driver?)
[ 5.289966] b308 16384 mmcblk3boot0
[ 5.289970] (driver?)
[ 5.296796] No filesystem could mount root, tried:
[ 5.296802] ext3
[ 5.301726] ext4
[ 5.303656] ext2
[ 5.305582] vfat
[ 5.307508] fuseblk
[ 5.309433]
[ 5.313137] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
[ 5.321412] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)

How can I access the initrd memory range from the secondary kernel to boot the board successfully?

@weidong_sun

Re: Kernel crash dump | kexec | kdump | initrd load issue issue

Zhiming_Liu — Mon, 07 Nov 2022 04:56:21 GMT

Please refer this https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/Kdump-Crash-in-I-MX/ta-p/1124321

Re: Kernel crash dump | kexec | kdump | initrd load issue issue

kevi49 — Mon, 07 Nov 2022 09:40:21 GMT

@Zhiming_Liu

Thanks for your reply,

We have followed the same document and we are able to load the secondary kernel as well but its is getting stuck in loading initrd,

crashkernel RAM execution is in 0x90000000-0x9fefffff address, But initrd is loaded in 0x83000000+0x00a44000 region and secondary kernel is not able access initrd and giving below error during boot.

INITRD: 0x83000000+0x00a44000 is not a memory region - disabling initrd

Can you please help in how to access the initrd from the secondary kernel so it can mount the file system and collect the vmcore dump.

Re: Kernel crash dump | kexec | kdump | initrd load issue issue

shiva_bhargav_y — Tue, 15 Aug 2023 13:12:00 GMT

Hi @kevi49 ,

how did you configured kdump, can help me with that on imx7.

thanks,

shiva.

i.MX ProcessorsのトピックRe: Kernel crash dump | kexec | kdump | initrd load issue issue

Kernel crash dump | kexec | kdump | initrd load issue issue

Re: Kernel crash dump | kexec | kdump | initrd load issue issue

Re: Kernel crash dump | kexec | kdump | initrd load issue issue

Re: Kernel crash dump | kexec | kdump | initrd load issue issue