Android 6 on i.MX 6QP - reboot problems with new chip revision 1.1

[bernhardfink]

A customer experiences the following problem on an i.MX 6DualPlus based product:

• Custom hardware assembled with 6QP rev 1.0 and rev 1.1 silicon. The only difference between the two revisions are changes in the boot ROM (security vulnerability fixes back in 2017)
• Android 6 with 3.14.52 kernel is running fine on both boards
• A reboot initiated from the command line works fine with rev1.0, but not with rev1.1
• Rebooting the board with rev1.1 silicon stops at the stage where the sdma firmware is loaded (doesn’t necessarily mean that the sdma firmware load is the reason for the crash), starts over and only succeeds in the second boot run
• When disabling/uninstalling a ttymxc1 service app, the reboot succeeds in the first run
• When removing the HDMI connection to a monitor, the reboot succeeds in the first run
• Both ttymxc1 and HDMI interface activity appear in the log shortly before/after the sdma firmware load

root@6qp:/ # reboot
SysRq : Emergency Remount R/O
EXT4-fs (sda6): re-mounted. Opts: (null)
EXT4-fs (sda7): re-mounted. Opts: (null)
EXT4-fs (sda4): re-mounted. Opts: (null)
Emergency Remount complete
sd 0:0:0:0: [sda] Synchronizing SCSI cache
imx2-wdt 20bc000.wdog: Device shutdown: Expect reboot!
reboot: Restarting system with command ''


U-Boot 2015.04 (Feb 09 2021 - 23:56:45)

CPU:   Freescale i.MX6DP rev1.0 at 792 MHz
CPU:   Temperature 48 C
Reset cause: WDOG
Board: custom
I2C:   ready
DRAM:  2 GiB
WARNING: Caches not enabled
No PMIC to init
MMC:   Warning: you configured more USDHC controllers(3) than supported by the board
FSL_SDHC: 0, FSL_SDHC: 1
AHCI 0001.0300 32 slots 1 ports 3 Gbps 0x1 impl SATA mode
flags: ncq stag pm led clo only pmp pio slum part
SATA Device Info:
S/N: E0118021500000001389
Product model number: 16GB SATA Flash Drive
Firmware version: SFDN004E
Capacity: 31277232 sectors
*** Warning - bad CRC, using default environment
 :
 :
Hit any key to stop autoboot:  0
boota sata
kernel   @ 14008000 (7788816)
ramdisk  @ 15000000 (1036591)
fdt      @ 14f00000 (34546)
## Booting Android Image at 0x12000000 ...
Kernel load addr 0x14008000 size 7607 KiB
Kernel command line: console=ttymxc0,115200 init=/init video=mxcfb0:dev=hdmi,bpp=32 video=mxcfb1:dev=lcd,bpp=32 video=mxcfb2:off video=mxcfb3:off vmalloc=256M androidboot.console=ttymxc0 consoleblank=0 androidboot.hardware=freescale cma=384M dmfc=3 androidboot.selinux=permissive
## Flattened Device Tree blob at 14f00000
   Booting using the fdt blob at 0x14f00000
   Loading Kernel Image ... OK
   Using Device Tree in place at 14f00000, end 14f0b6f1
LDO mode changes not supported 0

Starting kernel ...
 :
 :
imx-weim 21b8000.weim: Driver registered.
MIPI DSI driver module loaded
mxc_hdmi 20e0000.hdmi_video: Detected HDMI controller 0x13:0xa:0xa0:0xc1
fbcvt: 1920x1080@60: CVT Name - 2.073M9
mxc_sdc_fb fb.17: registered mxc display driver hdmi
Console: switching to colour frame buffer device 240x67
mxc_sdc_fb fb.18: registered mxc display driver lcd
mxc_hdmi 20e0000.hdmi_video: mxc_edid_read_internal
imx-sdma 20ec000.sdma: no iram assigned, using external mem
imx-sdma 20ec000.sdma: no event needs to be remapped
imx-sdma 20ec000.sdma: initialized
Serial: IMX driver
2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 58, base_baud = 5000000) is a IMX
imx-sdma 20ec000.sdma: loaded firmware 3.3

===> here the board starts over

U-Boot 2015.04 (Feb 09 2021 - 23:56:45)

CPU:   Freescale i.MX6DP rev1.1 at 792 MHz
CPU:   Temperature 58 C
Reset cause: WDOG
Board: custom
 :
 :

==> only in the second run it reaches the command prompt​

I can't test it by myself, because I only have 6QP boards with the rev1.0 silicon.

Any idea why the customer board with rev1.1 silicon starts over and only succeeds in the second boot run is welcome.

Thanks,
Bernhard.

[sumanthgavini]

Hi @bernhardfink @ceggers ,

On applying the above patch the imx6dp board not able to resolve the reboot behaviour. 

reboot behaviour( the imx6qp board booting into the kernel after the second boot, interrupted in the first boot by firmware loading) 

In the uboot the wdog is in a disabled state, No wdog configuration is specified in the uboot board-specific file. 

Note: If I power off/On the board(hard reset) the reboot behaviour is not observed. The reboot behaviour is only observed when I pass the reboot command from the kernel 

Could you provide further steps to resolve the reboot problem? 

[ceggers]

Hi Bernhard,

I cannot explain the difference in behavior between the two boards. But probably the problem is somehow related to the firmware loading procedure of the SDMA. I question whether the asynchronous firmware loading in sdma_get_firmware() is correct. Maybe there could be race conditions between the SDMA firmware loading and first DMA client drivers trying to request their channels (I didn't really investigate this). Your customer may try whether the following patch changes something:

--- a/drivers/dma/imx-sdma.c
+++ b/drivers/dma/imx-sdma.c
@@ -1889,11 +1889,11 @@
 static int sdma_get_firmware(struct sdma_engine *sdma,
 		const char *fw_name)
 {
-	int ret;
-
-	ret = request_firmware_nowait(THIS_MODULE,
-			FW_ACTION_HOTPLUG, fw_name, sdma->dev,
-			GFP_KERNEL, sdma, sdma_load_firmware);
+	struct firmware const *fw;
+	int ret = request_firmware(&fw, fw_name, sdma->dev);
+	if (ret)
+		return ret;
+	sdma_load_firmware(fw, sdma);
 
 	return ret;
 }

Another guess is, that the reset method may be somehow incomplete so that either the SDMA, AIPSTZ or one of the peripherals is in a non-reset state at the first reboot.

In the last lines of the log, U-Boot saya that the reset has been caused by the watchdog. Is the watchdog activated by the boot loader? Can the customer test again without a watchdog activated? I am curious whether the systems freezes or reboots anyway.

regards,
Christian

[bernhardfink]

Hi Christian,

thanks for your input. A conflict between the SDMA firmware loading and its first usage could explain the behavior, even if I have no idea why this just appears in the rev 1.1 silicon and then only on the first boot run. I need to compare again rev 1.0 silicon with 1.1, but I can't think of any impact on Linux loading.

The reset is another story, I have seen effects like this before, at least in MCU systems. The watchdog is used by Linux to initiate a reboot, I need to check if it's possible to disable it in u-boot and still do a reboot. I need to ask the customer if they can also generate a hardware reset, in order to compare the behavior with the watchdog reset.

Regards,
Bernhard.

[sumanthgavini]

@bernhardfink Are you able to recreate the reboot problem? Any clues to debug and resolve the issue