S32G, GoldVIP1.1.0, Xen v2xdomu, Kernel panic - not syncing

[AlanWuXuFeng]

Dear all,

After flushing the binary code of GlodVIP1.1.0 to a 32G SD-Card, power on the system, it looks well to active the Xen system. But when I switch the console to v2xdomu, it shows the following Kernel panic:

Auto Linux BSP 32.0 s32g274ardb2 hvc0

s32g274ardb2 login:
Auto Linux BSP 32.0 s32g274ardb2 ttyLF0

s32g274ardb2 login: root
Last login: Fri Mar 9 12:35:15 UTC 2018 on ttyLF0
root@s32g274ardb2:~# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 4 r----- 14.8
v2xdomu 1 1023 2 r----- 32.9
root@s32g274ardb2:~# xl console v2xdomu
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[ 0.000000] Linux version 5.10.90-rt60+g47a49b91d4d4 (oe-user@oe-host) (aarch64-fsl-linux-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35.1) #1 SMP PREEMPT Fri Mar 18 14:07:29 UTC 2022
[ 0.000000] Machine model: XENVM-4.14
[ 0.000000] Xen 4.14 support found
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000040000000-0x000000007fffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000040000000-0x000000007fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000007fffffff]
[ 0.000000] On node 0 totalpages: 262144
[ 0.000000] DMA zone: 4096 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 262144 pages, LIFO batch:63
[ 0.000000] cma: Reserved 256 MiB at 0x000000006e000000
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: Trusted OS migration not required
[ 0.000000] psci: SMC Calling Convention v1.1
[ 0.000000] percpu: Embedded 17 pages/cpu s32280 r8192 d29160 u69632
[ 0.000000] pcpu-alloc: s32280 r8192 d29160 u69632 alloc=17*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Detected VIPT I-cache on CPU0
[ 0.000000] CPU features: detected: ARM erratum 845719
[ 0.000000] CPU features: detected: GIC system register CPU interface
[ 0.000000] CPU features: detected: ARM errata 1165522, 1319367, 1530923, or 1530924
[ 0.000000] CPU features: detected: NXP erratum ERR050481 (TLBI handled incorrectly)
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 258048
[ 0.000000] Kernel command line: console=hvc0 root=/dev/xvda debug rw
[ 0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 751524K/1048576K available (9278K kernel code, 684K rwdata, 3020K rodata, 1472K init, 257K bss, 34908K reserved, 262144K cma-reserved)
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2.
[ 0.000000] Trampoline variant of Tasks RCU enabled.
[ 0.000000] Tracing variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] GICv3: 0 SPIs implemented
[ 0.000000] GICv3: 0 Extended SPIs implemented
[ 0.000000] GICv3: Distributor has no Range Selector support
[ 0.000000] GICv3: 16 PPIs implemented
[ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000003020000
[ 0.000000] random: get_random_bytes called from start_kernel+0x320/0x4c8 with crng_init=0
[ 0.000000] arch_timer: cp15 timer(s) running at 5.00MHz (virt).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x127350b88, max_idle_ns: 440795202120 ns
[ 0.000000] sched_clock: 56 bits at 5MHz, resolution 200ns, wraps every 4398046511100ns
[ 0.000369] Console: colour dummy device 80x25
[ 0.000406] printk: console [hvc0] enabled
[ 0.000436] Calibrating delay loop (skipped), value calculated using timer frequency.. 10.00 BogoMIPS (lpj=20000)
[ 0.000448] pid_max: default: 32768 minimum: 301
[ 0.000571] LSM: Security Framework initializing
[ 0.000675] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[ 0.000689] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[ 0.002748] xen:grant_table: Grant tables using version 1 layout
[ 0.002782] Grant table initialized
[ 0.002818] xen:events: Using FIFO-based ABI
[ 0.002868] Xen: initializing cpu0
[ 0.003031] rcu: Hierarchical SRCU implementation.
[ 0.003340] smp: Bringing up secondary CPUs ...
[ 0.003956] Detected VIPT I-cache on CPU1
[ 0.004006] GICv3: CPU1: found redistributor 1 region 0:0x0000000003040000
[ 0.010444] Xen: initializing cpu1
[ 0.010511] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[ 0.010684] smp: Brought up 1 node, 2 CPUs
[ 0.010691] SMP: Total of 2 processors activated.
[ 0.010698] CPU features: detected: 32-bit EL0 Support
[ 0.010703] CPU features: detected: CRC32 instructions
[ 0.025109] CPU: All CPU(s) started at EL1
[ 0.025159] alternatives: patching kernel code
[ 0.026436] devtmpfs: initialized
[ 0.027593] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.027612] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[ 0.035336] pinctrl core: initialized pinctrl subsystem
[ 0.036155] NET: Registered protocol family 16
[ 0.038039] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[ 0.038175] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[ 0.038323] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[ 0.038831] thermal_sys: Registered thermal governor 'step_wise'
[ 0.038939] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.039077] ASID allocator initialised with 65536 entries
[ 0.039185] Serial: AMBA PL011 UART driver
[ 0.045239] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.045254] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[ 0.045259] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.045264] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[ 0.047270] xen:balloon: Initialising balloon driver
[ 0.047806] vgaarb: loaded
[ 0.047997] SCSI subsystem initialized
[ 0.048191] usbcore: registered new interface driver usbfs
[ 0.048226] usbcore: registered new interface driver hub
[ 0.048260] usbcore: registered new device driver usb
[ 0.048406] pps_core: LinuxPPS API ver. 1 registered
[ 0.048412] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 0.048425] PTP clock support registered
[ 0.049591] clocksource: Switched to clocksource arch_sys_counter
[ 0.059336] NET: Registered protocol family 2
[ 0.059482] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 0.060447] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
[ 0.060475] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.060580] TCP bind hash table entries: 8192 (order: 5, 131072 bytes, linear)
[ 0.060713] TCP: Hash tables configured (established 8192 bind 8192)
[ 0.060821] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[ 0.060853] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[ 0.061005] NET: Registered protocol family 1
[ 0.061694] RPC: Registered named UNIX socket transport module.
[ 0.061700] RPC: Registered udp transport module.
[ 0.061704] RPC: Registered tcp transport module.
[ 0.061707] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.061743] PCI: CLS 0 bytes, default 64
[ 0.062748] Initialise system trusted keyrings
[ 0.062961] workingset: timestamp_bits=46 max_order=18 bucket_order=0
[ 0.063946] fuse: init (API version 7.32)
[ 0.064218] NET: Registered protocol family 38
[ 0.064230] Key type asymmetric registered
[ 0.064235] Asymmetric key parser 'x509' registered
[ 0.064264] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
[ 0.064271] io scheduler mq-deadline registered
[ 0.064276] io scheduler kyber registered
[ 0.071581] xen:xen_evtchn: Event-channel device installed
[ 0.073386] serial: Freescale linflex driver
[ 0.073836] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 0.073873] Invalid max_queues (4), will use default max: 2.
[ 0.077835] libphy: Fixed MDIO Bus: probed
[ 0.078171] vcan: Virtual CAN interface driver
[ 0.078176] slcan: serial line CAN interface driver
[ 0.078178] slcan: 10 dynamic interface channels.
[ 0.078185] CAN device driver interface
[ 0.078241] e100: Intel(R) PRO/100 Network Driver
[ 0.078244] e100: Copyright(c) 1999-2006 Intel Corporation
[ 0.078292] e1000: Intel(R) PRO/1000 Network Driver
[ 0.078296] e1000: Copyright (c) 1999-2006 Intel Corporation.
[ 0.078330] e1000e: Intel(R) PRO/1000 Network Driver
[ 0.078333] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[ 0.078365] igb: Intel(R) Gigabit Ethernet Network Driver
[ 0.078368] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 0.078500] xen_netfront: Initialising Xen virtual ethernet driver
[ 0.088605] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 0.088620] ehci-pci: EHCI PCI platform driver
[ 0.088962] usbcore: registered new interface driver uas
[ 0.089002] usbcore: registered new interface driver usb-storage
[ 0.089563] i2c /dev entries driver
[ 0.089803] sdhci: Secure Digital Host Controller Interface driver
[ 0.089806] sdhci: Copyright(c) Pierre Ossman
[ 0.089808] sdhci-pltfm: SDHCI platform and OF driver helper
[ 0.089977] usbcore: registered new interface driver usbhid
[ 0.089982] usbhid: USB HID core driver
[ 0.090843] NET: Registered protocol family 10
[ 0.092054] Segment Routing with IPv6
[ 0.092138] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 0.092741] NET: Registered protocol family 17
[ 0.092802] Bridge firewalling registered
[ 0.097781] can: controller area network core
[ 0.097882] NET: Registered protocol family 29
[ 0.097892] can: raw protocol
[ 0.097898] can: broadcast manager protocol
[ 0.097908] can: netlink gateway - max_hops=1
[ 0.098573] 8021q: 802.1Q VLAN Support v1.8
[ 0.098977] printk: console [hvc0]: printing thread started
[ 0.099036] Loading compiled-in X.509 certificates
[ 0.138654] xen_netfront: backend supports XDP headroom
[ 0.149230] random: fast init done
[ 0.192956] xen_netfront: backend supports XDP headroom
[ 0.213033] blkfront: xvda: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[ 0.335330] List of all partitions:
[ 0.335358] ca00 6217728 xvda
[ 0.335366] driver: vbd
[ 0.335370] No filesystem could mount root, tried:
[ 0.335371] ext3
[ 0.335374] ext4
[ 0.335375] ext2
[ 0.335378] vfat
[ 0.335379] fuseblk
[ 0.335381]
[ 0.335383] Kernel panic - not syncing:

So what is the matter to my current system?

I also checked the disk partitions:

root@s32g274ardb2:~# cat /proc/partitions
major minor #blocks name

179 0 30948352 mmcblk0
179 1 65536 mmcblk0p1
179 2 2097152 mmcblk0p2
179 3 6217728 mmcblk0p3
179 4 1 mmcblk0p4
179 5 6217728 mmcblk0p5
root@s32g274ardb2:~#

it looks all of the partitions are well configured.

Thanks.

Alan

[nxf45013]

Hi, Alan, 

The partitioning is done at boot time of Domain0 (Through this, we reduced the SD Card Size). 

Please run this command from Domain0, this might fix the configuration issues: e2fsck -pf /dev/mmcblk0p

Did you somehow reset the board right after first login? Could you please try with a different card to rule this out?  

Vlad 

[AlanWuXuFeng]

Hi, Vlad,

I think you mean the command:

e2fsck -pf /dev/mmcblk0p3

for the V2Xdomu.cfg setup the physical storage device to V2Xdomu as

disk = [ 'phy:/dev/mmcblk0p3,xvda,w' ]

but it does not work and shows the following message:

root@s32g274ardb2:~# e2fsck -pf /dev/mmcblk0p3
e2fsck: Bad magic number in super-block while trying to open /dev/mmcblk0p3
/dev/mmcblk0p3:
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>

Same as /dev/mmcblk0p5.  It seems the partition has been broken.

I would like to try another 16GB ScanDisk SD card, do you think this storage size is enough for the whole system?  for it shows the error during the dd processing:

dd: error writing '/dev/sdb': No space left on device

Why we need to setup /dev/mmcblk0p4 and /dev/mmcblk0p5 on target ?

By the way, it there some way to fix the SD partition issue in our host Linux PC?  for example, after the first target system power on based on a new flushed SD card, we can wait the target system to figure out its internal partitions by itself, and then power off the target, plug out the SD card to the host Linux PC and do some fixing there.  I guess the current xvda filesystem is same as ext4.

Regards

Alan

[gmocanu]

Hello, Alan,

A 16GB SD card should be enough to fit the whole system. The size of the GoldVIP 1.1.0 .sdcard image is approximately 3.5GiB, so there should be no problem in writing it on any SD card with a capacity greater or equal to 16GB (an 8GB SD card may also work, but there will be a limitation with the OTA updates use case). Can you try to write it using a different tool (i.e., Win32diskimager from Windows)?

The visible partition table from the GoldVIP 1.1.0 .sdcard image contains:

• mmcblk0p1 - the boot partition, containing the Linux kernel image, .dtb, etc.
• mmcblk0p2 - the dom0 rootfs partition, set to a size of 2 GiB
• mmcblk0p3 - the v2xdomu rootfs partition, which is extended during the first boot (before the VM is started)

During the first boot, the partition table is altered to use as much space available on the SD card as possible and create the other partition required for the OTA updates use case (mmcblk0p5). mmcblk0p4 is just an extended partition that allows the developers to further format and use the remaining disk space.

The v2xdomu partition is formatted as ext4, and xvda is the block device seen by the guest (in v2xdomu console the xvda device is mounted as /).

Could you re-write the image on the same SD card and boot up the platform again? Is the same behavior still observed? Could you try to use another SD card (maybe the current one is close to its end?)? If you can reproduce the behavior, do you mind sharing the console log shown during the first boot? It may contain useful information about what happens in there.

Best regards,

George

[AlanWuXuFeng]

Hello George,

According to your message and suggestion, my current solution works as followed:

1. Flush the code into a new SD Card. ( I kept the one with 32GByte to expand other application)

2. Power on the system with this new flushed SD Card at first time.

3. After the shell system is ready for  Xen domain0, do some fixing for mmcblock0p3

e2fsck -pf /dev/mmcblk0p3

4. Double check the v2xdomu by

xl console v2xdomu 

it looks well with kinds of application there.

Let's do some further research.

Thanks and best regards.

Alan

[gmocanu]

Hello, Alan,

I understand that you have to run the e2fsck command after the first boot. What happens if you don't run it? Is the same behavior observed? Could you try to re-flash the SD card and just try to login to v2xdomu VM without executing any prior command? If the v2xdomu filesystem is corrupted, could you attach the dmesg logs, please?

It would be a bit strange if you have to run it every time since the same command is executed while the partition table is altered [1].

Thanks,

George

[1] - https://source.codeaurora.org/external/autobsps32/goldvip/gvip/tree/sdcard-partitioning/service/sdca...