Failure to boot i.MX28 after suspend

[thekarl]

The company I work at has developed a custom board where the main processor is an i.MX28, which runs Linux. On the latest revision of the board we have run into serious problems after the system has been brought down to a suspend state. After this it's no longer possible to boot the system, not even after resetting the power. Network booting and downloading new software with MfgTool also fail.

We need some help in diagnosing what has happened, since earlier revisions of the board don't have this problem. Differences between the board revisions that might be relevant:

• The circuitry driving the PSWITCH pin has been changed. It's more complicated for the latest revision.
• Different revisions of flash and RAM memory chips. Might be relevant since MfgTool fails to flash new software, once the issue has taken hold

The circuitry driving the PSWITCH pin on the latest revision of the board has not been tested properly. It is possible that the voltage on the PSWITCH pin spikes to unallowed values. But could this cause the issues described below?

Procedure leading up to the problem:

• Flashing software with MfgTool works fine for a fresh board.
• Booting the system also works fine and our application works as expected.
• After resetting the power the system boots normally.
• The problem seems to occur when putting the system to suspend (our application writes "standby" to /sys/power/state). This is triggered by an external switch which also brings the voltage on the PSWITCH pin to 0V.
• Resetting the switch brings up the PSWITCH voltage. The CPU does not resume operation. Nothing is printed to the console. There might be problems with the PSWITCH circuitry here, but it seems strange if that still has effect after resetting the power
• Resetting the Power leads to the inability to boot the system properly. See the console outputs below

Output when booting

HTLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLFLC
PowerPrep start initialize power...

Configured for 5v only power source.            Battery powered operation disabled.
LLLCSep  9 201518:04:54
FRAC 0x92925552
memory type is DDR2
                   Wait for ddr ready 1power 0x00820616
Frac 0x92925552
start change cpu freq
hbus 0x00000003
cpu 0x00010001
start test memory accress
ddr2 0x40000000
finish simple test
LLLLLFLCLLJUncompressing Linux...

uncompression error

-- System halted

Output when network booting (SD-card with u-boot)

PowerPrep start initialize power...

Configured for 5v only power source.            Battery powered operation disabled.
Jun 14 201215:39:08
FRAC 0x92925552
memory type is DDR2
                   Wait for ddr ready 1power 0x00820616
Frac 0x92925552
start change cpu freq
hbus 0x00000003
cpu 0x00010001
start test memory accress
ddr2 0x40000000
finish simple test

U-Boot 2009.08-dirty (Feb 14 2012 - 12:57:14)

Freescale i.MX28 family
CPU:   454 MHz
BUS:   151 MHz
EMI:   205 MHz
GPMI:   24 MHz
DRAM:  128 MB
MMC:   IMX_SSP_MMC: 0, IMX_SSP_MMC: 1
*** Warning - bad CRC or MMC, using debault enviro.ment

Note the corrupted characters at the last line (using debault enviro.ment). After the last line is printed, nothing further happens. The same SD-card work fine in a board which has not suffered the error yet.

Flashing software with MfgTool

Below is the contents of the MfgTool log and the ucl.xml file.  MfgTool fails at the second command (<CMD type="find" body="Updater" timeout="180"/>). The error code is -65536, which I don't know what it means.

mfgtool.log:

Monday, May 23, 2016 09:13:03
1 - Panel A Start processing Complete <LIST/>.

1 - Panel A Start <CMD/> type="boot" body="Recovery" file="updater_ivt.sb" timeout="60" onError="" text="Booting update firmware.".

1 - Panel A Finished <CMD/> type="boot" body="Recovery" file="updater_ivt.sb" timeout="60" onError="" text="Booting update firmware." SUCCESS code=0.

1 - Panel A Start <CMD/> type="find" body="Updater" file="" timeout="180" onError="" text="".

1 - Panel A Finished <CMD/> type="find" body="Updater" file="" timeout="180" onError="" text="" FAIL code=-65536.

Panel A: Updater Error 0xffff0000 (-65536) -

2 - Panel A Finished processing Complete <LIST/> : FAIL code=-65536.

ucl.xml

<!--
* Copyright (C) 2010, Freescale Semiconductor, Inc. All Rights Reserved.
* The CFG element contains a list of recognized usb devices.
*  DEV elements provide a name, class, vid and pid for each device.
*
* Each LIST element contains a list of update instructions.
*  "Install" - Erase media and install firmware.
*  "Update" - Update firmware only.
*
* Each CMD element contains one update instruction of attribute type.
*  "pull" - Does UtpRead(body, file) transaction.
*  "push" - Does UtpWrite(body, file) transaction.
*  "drop" - Does UtpCommand(body) then waits for device to disconnect.
*  "boot" - Finds configured device, forces it to "body" device and downloads "file".
*  "find" - Waits for "timeout" seconds for the "body" device to connect.
*  "show" - Parse and show device info in "file". 
*<STATE name="Recovery" dev="IMX28"/>
*<DEV name="IMX28" vid="15A2" pid="004F"/>
*
* FIXME:
* Currently, the sd image can use with uboot or without uboot
* Nand image not uses uboot (only linux.sb)
*
-->

<UCL>
  <CFG>
    <STATE name="Recovery" dev="IMX28"/>
    <STATE name="Updater"   dev="Updater" /> 
    <DEV name="IMX28" vid="15A2" pid="004F"/>
    <DEV name="Updater" vid="066F" pid="37FF" />
  </CFG>

<LIST name="Complete" desc="Install on singlechip NAND">
<CMD type="boot" body="Recovery" file="updater_ivt.sb" timeout="60" if="HabEnable">Booting update firmware.</CMD>
<CMD type="find" body="Updater" timeout="180"/>
<CMD type="push" body="mknod class/mtd,mtd0,/dev/mtd0"/>
<CMD type="push" body="mknod class/mtd,mtd1,/dev/mtd1"/>
<CMD type="push" body="mknod class/misc,ubi_ctrl,/dev/ubi_ctrl"/>
<CMD type="push" body="$ flash_eraseall /dev/mtd0">Erasing rootfs partition</CMD>
<CMD type="push" body="$ flash_eraseall /dev/mtd1">Erasing rootfs partition</CMD>
<CMD type="push" body="send" file="files/imx28_ivt_linux.sb">Sending firmware</CMD>
<CMD type="push" body="$ kobs-ng init $FILE">Flashing firmware</CMD>
<CMD type="push" body="pipe ubiformat /dev/mtd1 -f - -S @FILESIZE" file="files/rootfs.ubifs.img">Transfer rootfs</CMD>
<CMD type="push" body="frf">Finish Flashing NAND</CMD>

<CMD type="push" body="mknod class/mtd,mtd0,/dev/mtd0"/>
<CMD type="push" body="mknod class/mtd,mtd1,/dev/mtd1"/>
<CMD type="push" body="mknod class/misc,ubi_ctrl,/dev/ubi_ctrl"/>

<CMD type="push" body="$ ubiattach /dev/ubi_ctrl -m 1">Attaching NAND</CMD>
<CMD type="push" body="$ mkdir -p /mnt/nand">Mounting rootfs</CMD>
<CMD type="push" body="$ mount -t ubifs ubi0:rootfs /mnt/nand">Mounting rootfs</CMD>

<CMD type="push" body="send" file="files/vagsag-ioc.ini">Transfering ioc</CMD>
<CMD type="push" body="$ cp $FILE /mnt/nand/ioc/vagsag-ioc.ini" file="files/vagsag-ioc.ini">Transfering ioc</CMD>
<CMD type="push" body="send" file="files/vagsag-ioc.s19">Transfering ioc</CMD>
<CMD type="push" body="$ cp $FILE /mnt/nand/ioc/vagsag-ioc.s19" file="files/vagsag-ioc.s19">Transfering ioc</CMD>

<CMD type="push" body="$ umount /mnt/nand">Unmounting rootfs</CMD>
<CMD type="push" body="$ ubidetach /dev/ubi_ctrl -m1">Unmounting rootfs</CMD>

<CMD type="push" body="frf">Finish updating IOC</CMD>
<CMD type="push" body="$ echo Update Complete!">Done</CMD>

<!-- Enable below to reboot device
<CMD type="push" body="!3">Done</CMD>
-->

</LIST>

<LIST name="IOC Only" desc="Install on singlechip NAND">
<CMD type="boot" body="Recovery" file="updater_ivt.sb" timeout="60" if="HabEnable">Booting update firmware.</CMD>
<CMD type="find" body="Updater" timeout="180"/>

<CMD type="push" body="mknod class/mtd,mtd0,/dev/mtd0"/>
<CMD type="push" body="mknod class/mtd,mtd1,/dev/mtd1"/>
<CMD type="push" body="mknod class/misc,ubi_ctrl,/dev/ubi_ctrl"/>

<CMD type="push" body="$ ubiattach /dev/ubi_ctrl -m 1">Attaching NAND</CMD>
<CMD type="push" body="$ mkdir -p /mnt/nand">Mounting rootfs</CMD>
<CMD type="push" body="$ mount -t ubifs ubi0:rootfs /mnt/nand">Mounting rootfs</CMD>

<CMD type="push" body="send" file="files/vagsag-ioc.ini">Transfering ioc</CMD>
<CMD type="push" body="$ cp $FILE /mnt/nand/ioc/vagsag-ioc.ini" file="files/vagsag-ioc.ini">Transfering ioc</CMD>
<CMD type="push" body="send" file="files/vagsag-ioc.s19">Transfering ioc</CMD>
<CMD type="push" body="$ cp $FILE /mnt/nand/ioc/vagsag-ioc.s19" file="files/vagsag-ioc.s19">Transfering ioc</CMD>

<CMD type="push" body="$ umount /mnt/nand">Unmounting rootfs</CMD>
<CMD type="push" body="$ ubidetach /dev/ubi_ctrl -m1">Unmounting rootfs</CMD>

<CMD type="push" body="frf">Finish updating IOC</CMD>

<CMD type="push" body="$ echo Update Complete!">Done</CMD>

<!-- Enable below to reboot device
<CMD type="push" body="!3">Done</CMD>
-->
</LIST>

</UCL>

[b36401]

In this log I see that Linux loader starts fine and then it fails to unpack the kernel into RAM.

So I suppose it can be caused with inaccuracy memory design.

Please try to run DDR stress test on the board.

Have a great day,

Victor

-----------------------------------------------------------------------------------------------------------------------

Note: If this post answers your question, please click the Correct Answer button. Thank you!

-----------------------------------------------------------------------------------------------------------------------

[thekarl]

Thanks for the reply!

Is there a specific DDR stress test you're thinking about? I found a test in another thread (i.MX28 DDR stress test)​ called mem_test, but I'm having trouble downloading it to the device. The problem, however, seems to be related to sb_loader rather than the test.

Running sb_loader.exe /f test.sb gives this:

Downloading test.sb to device.

CStHidDevice::Download()  Error(-13) during Write cbw.

Quitting.

  Error(-13) during download.

Quitting.

The same output is also written when I try to download other files. What does the -13 error code mean?