I.MX6SoloX2 - Linux 5.4.70_2.3.0 - Kernel hangs while booting without any error

[parth_shah]

Hello folks,

I have been using Linux 4.14.238 version for I.MX6SoloX2 SOM without any issues. Due dependency of custom network switch driver which is supported on Linux 5.4, I have to move the Linux kernel version from 4.14 to 5.4

I am using NXP release of 5.4.70_2.3.0 supported for i.MX6SoloX Processor. I am able to successfully compile the compile the packages. 

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

https://source.codeaurora.org/external/imx/imx-manifest/tree/README?h=imx-linux-zeus

But When the Board boots up, without showing any error kernel hangs indefinitely after loading ahci (SATA) driver.

I checked imx6sx.dtsi file for both linux versions and could not find major difference which could lead to this error. I have been using default kernel configuration imx_v7_defconfig.

Can anyone please share their opinion what could be reason of this error?

## Booting kernel from Legacy Image at 82000000 ...
Image Name: Linux-5.4.70+g05ce692fbc29
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 9015880 Bytes = 8.6 MiB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum ... OK
## Flattened Device Tree blob at 81000000
Booting using the fdt blob at 0x81000000
Loading Kernel Image ... OK
Loading Device Tree to 9df1b000, end 9df28e0e ... OK

Starting kernel ...

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.4.70+g05ce692fbc29 (oe-user@oe-host) (gcc version 9.3.0 (GCC)) #1 SMP PREEMPT Wed Dec 16 05:31:24 UTC 2020
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: Freescale i.MX6 SoloX SG base Board
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] Reserved memory: created CMA memory pool at 0x88000000, size 320 MiB
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[ 0.000000] percpu: Embedded 15 pages/cpu s31244 r8192 d22004 u61440
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 129666
[ 0.000000] Kernel command line: console=ttymxc0,115200 ubootver=2016.03-g25d39d31-g7969642 root=/dev/mmcblk3p2 rw rootfstype=ext4
[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 171316K/523264K available (11264K kernel code, 533K rwdata, 4464K rodata, 1024K init, 435K bss, 24268K reserved, 327680K cma-reserved, 0K highmem)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=1.
[ 0.000000] Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] L2C-310 erratum 769419 enabled
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[ 0.000000] L2C-310 ID prefetch enabled, offset 16 lines
[ 0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 256 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76430001
[ 0.000000] random: get_random_bytes called from start_kernel+0x2c8/0x488 with crng_init=0
[ 0.000000] Switching to timer-based delay loop, resolution 333ns
[ 0.000007] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 715827882841ns
[ 0.000032] clocksource: mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 637086815595 ns
[ 0.001468] Console: colour dummy device 80x30
[ 0.001496] Calibrating delay loop (skipped), value calculated using timer frequency.. 6.00 BogoMIPS (lpj=30000)
[ 0.001511] pid_max: default: 32768 minimum: 301
[ 0.001664] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.001678] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.002424] CPU: Testing write buffer coherency: ok
[ 0.002467] CPU0: Spectre v2: using BPIALL workaround
[ 0.002756] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.003352] Setting up static identity map for 0x80100000 - 0x80100060
[ 0.003524] rcu: Hierarchical SRCU implementation.
[ 0.003814] smp: Bringing up secondary CPUs ...
[ 0.003826] smp: Brought up 1 node, 1 CPU
[ 0.003835] SMP: Total of 1 processors activated (6.00 BogoMIPS).
[ 0.003842] CPU: All CPU(s) started in SVC mode.
[ 0.004220] devtmpfs: initialized
[ 0.009535] Duplicate name in lcdif@2224000, renamed to "display#1"
[ 0.011327] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.011727] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.011750] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.032371] pinctrl core: initialized pinctrl subsystem
[ 0.033385] NET: Registered protocol family 16
[ 0.040160] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.049611] vdd1p1: supplied by regulator-dummy
[ 0.050169] vdd3p0: supplied by regulator-dummy
[ 0.050782] vdd2p5: supplied by regulator-dummy
[ 0.051323] vddarm: supplied by regulator-dummy
[ 0.051893] vddpcie: supplied by regulator-dummy
[ 0.052430] vddsoc: supplied by regulator-dummy
[ 0.053099] mxs_phy 20c9000.usbphy: 20c9000.usbphy supply phy-3p0 not found, using dummy regulator
[ 0.053623] mxs_phy 20ca000.usbphy: 20ca000.usbphy supply phy-3p0 not found, using dummy regulator
[ 0.067235] No ATAGs?
[ 0.067299] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.067313] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.069596] imx6sx-pinctrl 20e0000.iomuxc: Invalid fsl,pins or pins property in node /soc/aips-bus@2000000/iomuxc@20e0000/imx6x-sg/pciereggrp
[ 0.069686] imx6sx-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
[ 0.072084] imx mu driver is registered.
[ 0.073071] imx rpmsg driver is registered.
[ 0.074379] MU is ready for cross core communication!
[ 0.103077] vgaarb: loaded
[ 0.103939] SCSI subsystem initialized
[ 0.104442] usbcore: registered new interface driver usbfs
[ 0.104495] usbcore: registered new interface driver hub
[ 0.104581] usbcore: registered new device driver usb
[ 0.104752] usb_phy_generic usbphynop1: usbphynop1 supply vcc not found, using dummy regulator
[ 0.106392] i2c i2c-0: IMX I2C adapter registered
[ 0.107320] i2c i2c-1: IMX I2C adapter registered
[ 0.108083] pca953x 2-0020: using no AI
[ 0.131682] pca953x 2-0021: using no AI
[ 0.140833] pca953x 2-0021: failed writing register
[ 0.140989] i2c i2c-2: IMX I2C adapter registered
[ 0.141869] i2c i2c-3: IMX I2C adapter registered
[ 0.142394] mc: Linux media interface: v0.10
[ 0.142446] videodev: Linux video capture interface: v2.00
[ 0.142500] pps_core: LinuxPPS API ver. 1 registered
[ 0.142509] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 0.142529] PTP clock support registered
[ 0.170784] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 0.172291] MIPI CSI2 driver module loaded
[ 0.172346] Advanced Linux Sound Architecture Driver Initialized.
[ 0.173521] Bluetooth: Core ver 2.22
[ 0.173580] NET: Registered protocol family 31
[ 0.173589] Bluetooth: HCI device and connection manager initialized
[ 0.173606] Bluetooth: HCI socket layer initialized
[ 0.173616] Bluetooth: L2CAP socket layer initialized
[ 0.173641] Bluetooth: SCO socket layer initialized
[ 0.174252] clocksource: Switched to clocksource mxc_timer1
[ 0.174435] VFS: Disk quotas dquot_6.6.0
[ 0.174521] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 0.183743] thermal_sys: Registered thermal governor 'step_wise'
[ 0.184070] NET: Registered protocol family 2
[ 0.184729] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[ 0.184769] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[ 0.184815] TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.184882] TCP: Hash tables configured (established 4096 bind 4096)
[ 0.184978] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 0.185007] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 0.185210] NET: Registered protocol family 1
[ 0.185773] RPC: Registered named UNIX socket transport module.
[ 0.185786] RPC: Registered udp transport module.
[ 0.185793] RPC: Registered tcp transport module.
[ 0.185801] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.186339] PCI: CLS 0 bytes, default 64
[ 0.186996] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 0.188716] Bus freq driver module loaded
[ 0.189728] Initialise system trusted keyrings
[ 0.189955] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[ 0.196323] NFS: Registering the id_resolver key type
[ 0.196354] Key type id_resolver registered
[ 0.196362] Key type id_legacy registered
[ 0.196407] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[ 0.196911] fuse: init (API version 7.31)
[ 0.232248] Key type asymmetric registered
[ 0.232262] Asymmetric key parser 'x509' registered
[ 0.232285] io scheduler mq-deadline registered
[ 0.232293] io scheduler kyber registered
[ 0.247385] imx-sdma 20ec000.sdma: Direct firmware load for imx/sdma/sdma-imx6q.bin failed with error -2
[ 0.247403] imx-sdma 20ec000.sdma: Falling back to sysfs fallback for: imx/sdma/sdma-imx6q.bin
[ 0.249447] mxs-dma 1804000.dma-apbh: initialized
[ 0.256721] 2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 24, base_baud = 5000000) is a IMX
[ 1.043776] printk: console [ttymxc0] enabled
[ 1.049225] 21e8000.serial: ttymxc1 at MMIO 0x21e8000 (irq = 68, base_baud = 5000000) is a IMX
[ 1.058813] 21ec000.serial: ttymxc2 at MMIO 0x21ec000 (irq = 69, base_baud = 5000000) is a IMX
[ 1.068304] 21f0000.serial: ttymxc3 at MMIO 0x21f0000 (irq = 70, base_baud = 5000000) is a IMX
[ 1.077769] 21f4000.serial: ttymxc4 at MMIO 0x21f4000 (irq = 71, base_baud = 5000000) is a IMX
[ 1.087265] 22a0000.serial: ttymxc5 at MMIO 0x22a0000 (irq = 83, base_baud = 5000000) is a IMX
[ 1.098050] imx sema4 driver is registered.
[ 1.119720] brd: module loaded
[ 1.131392] loop: module loaded
[ 1.135889] at24 1-0052: 4096 byte 24c32 EEPROM, writable, 32 bytes/write
[ 1.144203] imx ahci driver is registered.

[parth_shah · Accepted solution]

In dts node, I set the SPI max Frequency as 40 MHz(spi-max-frequency = <40000000>). Due to some change in the driver, it gets multiplied with 4, hence SPI frequency becomes 160MHz. 

So, setting SPI max frequency as 10 MHz (spi-max-frequency = <10000000>) solves the problem.

[igorpadykov]

Hi Parth

one can check if proper uboot version was used for Linux 5.4.70_2.3.0  :

https://source.codeaurora.org/external/imx/uboot-imx/tree/?h=imx_v2020.04_5.4.70_2.3.0

also one can try to decrease cma size

https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm/boot/dts/imx6sx.dtsi?h=imx_5.4.70...

Best regards
igor

[parth_shah]

I found out that it gets stuck because of QSPI node.

&qspi2 {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_qspi2>;
status = "okay";
ddrsmp=<0>; //or 2

flash0: w25q128@0 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "winbond,w25q128", "jedec,spi-nor";
m25p,fast-read;
spi-max-frequency = <40000000>;
reg = <0>;
};
};

If I disable this node, kernel boots, but it gets stuck elsewhere.

Could you please check if configuration is proper or not?

[igorpadykov]

>Could you please check if configuration is proper or not?

one can look at nxp official imx_v2020.04_5.4.70_2.3.0 uboot

https://source.codeaurora.org/external/imx/uboot-imx/tree/arch/arm/dts/imx6sx-sdb.dts?h=imx_v2020.04...

Best regards
igor

[parth_shah]

I have same dts entry, still does not bootup

[igorpadykov]

are you using imx_v2020.04_5.4.70_2.3.0 or other ?

Best regards
igor

[parth_shah]

Yes, I am using  imx_v2020.04_5.4.70_2.3.0 for i.MX6SoloX2

[igorpadykov]

one can try to debug it and try to find reason for halt using  AN4553 Using Open Source Debugging Tools for Linux on i.MX Processors
https://www.nxp.com/docs/en/application-note/AN4553.pdf

Best regards
igor