Nand problems - mx6ul commercial to industrial

[grantsinclair]

Hi

I'm working with an i.mx6ul on a custom board. Recently I've moved from the commercial part to the industrial part (MCIMX6G2DVM05AA to MCIMX6G2CVM05AB) but now I'm having NAND problems at boot (see log below). Sometimes I'm able to load an image to nand, but then on subsequent boots I get the same NAND read error. Erasing the NAND from u-boot sometimes reports problems too.

I2C: ready
DRAM: 256 MiB
NAND: 128 MiB
MMC: FSL_SDHC: 0, FSL_SDHC: 1
MXS NAND: DMA read error
NAND read from offset c80000 failed -110

I'm aware of errata 11115, but the parts placed have date code 1831 so should not be susceptible to this. I had assumed the commercial and industrial versions would behave identically - I haven't found any documented differences. There have been some small copper changes to the PCB layout, but these look low-risk and there are no changes to any other components.

Thanks and kind regards

Grant Sinclair

[grantsinclair]

Hi Igor

Thanks for your help and advice.

I have some more information, and I'd like to know what you think please.

Board #1

With boot config set for NAND, I've never been able to load images using MFG tools (the NAND is never mounted as a mass storage device). 

So I changed to boot from SD card and I have now got further. At room temperature, uboot shows no NAND:

I2C: ready
DRAM: 256 MiB
NAND: 0 MiB
MMC: FSL_SDHC: 0, FSL_SDHC: 1

This is confirmed using "nand devices" which returns "no devices found".

I heated the board to 55 degrees C and this changed - NAND was now available and successfully read every boot.

Then I cooled the board to 4 degrees and the board stopped running very early in uboot.

After both heating and cooling I let the board return to room temperature - and at room temperature it still behaved as previously, ie no NAND found.

Board #2

When boot config set for NAND, I can load images using the MFG tools. But then afterwards trying to boot normally I get NAND DMA errors (as I've previously mentioned). Also doing "nand erase.chip" from uboot shows multiple errors. I wondered about slowing the boot frequencies, so I tied BOOT_CFG2[2] (LCD1_DATA10) high. I've verified this works by reading the shadow register from uboot (md 0x20d8004) and comparing with previous. It didn't make any difference to the performance of the board, but the uboot "clocks" command returned numbers that seem strange (eg DDR isn't 200MHz):

PLL_SYS 792 MHz
PLL_BUS 528 MHz
PLL_OTG 480 MHz
PLL_NET 50 MHz

IPG 38322 kHz
UART 80000 kHz
AHB 76645 kHz
AXI 153290 kHz
DDR 306580 kHz
USDHC1 198000 kHz
USDHC2 198000 kHz
USDHC3 0 kHz
USDHC4 0 kHz
EMI SLOW 76645 kHz
IPG PERCLK 38322 kHz

compared to another board with BOOT_CFG2[2] low:

PLL_SYS 792 MHz
PLL_BUS 528 MHz
PLL_OTG 480 MHz
PLL_NET 50 MHz

IPG 66000 kHz
UART 80000 kHz
AHB 132000 kHz
AXI 198000 kHz
DDR 396000 kHz
USDHC1 198000 kHz
USDHC2 198000 kHz
USDHC3 0 kHz
USDHC4 0 kHz
EMI SLOW 99000 kHz
IPG PERCLK 66000 kHz

Many thanks

Grant

[igorpadykov]

Hi Grant

changing behavior with temperature may point to signal integrity

issues, also this may point to soldering issues.

May be recommended to rerun ddr test for each board and update image with new

ddr calbration settings, check clocks (32.768KHz, 24 MHz) stability and power supplies

ripples described in Hardware Development Guide for the i.MX 6UltraLite Applications Processor

Best regards
igor

[grantsinclair]

Hi Igor

Thanks for your continued support.

I've adjusted the NAND DSE for all the pads, with no difference. It was R0/6, so I've tried R0/7, R0/5, and R0/4. With R0/4, I reliably got "cannot support the NAND, missing necessary info" from uboot, so I didn't go any further.

I've also checked power supply voltage levels and noise, and verified the 32k and 24M crystals. I've scoped some of the NAND signals (CE, RE, WE, READY) and they look clean.

I've verified that the boot configuration is set as I think it should be (using gpio) and verified the shadow registers match. Also checked no efuses have been blown.

POR is de-asserted after power supplies have comeup.

I've also asked our hardware engineer to look over the hardware development guide, and he hasn't found anything to note.

What does it mean that uboot can be loaded from NAND and runs ok, but nothing after that? From within uboot, if I stop auto boot and execute "nand dump -o -a /dev/mtd0", I get useful data about 50% of the time.

Best regards

Grant

[igorpadykov]

Hi Grant

just for test one can try to heat/cold processor, also if this one/two board

this may be soldering issues.

In general one can test nand in linux:

How-To use NAND boot on i.MX6UL EVK board 

Using MFG TOOLS with i.MX 6UltraLite EVK board 

Best regards
igor

[grantsinclair]

Hi Igor

Thanks for your reply. I've run the DDR stress test and updated the image - but no change to the problem. I ran the stress test tool from uboot 7 times and had 4 successful runs. 1 time it failed (ERROR FOUND, we can't get suitable value !!!!
dram test fails for all values) and another 2 times it just stopped and didn't complete. The results from the 4 successful results appeared to be consistent.

This problem seems to show up in other ways too:

//After loading images via manufacturing tools, then reboot, stop auto boot, nand erase.chip:
NAND erase.chip: device 0 whole chip
Erasing at 0x1d60000 -- 23% complete.
nand0: MTD Erase failure: -5
Erasing at 0x7fe0000 -- 100% complete.
OK

-------------------------

//On boot after loading manufacturing tools image

I2C: ready
DRAM: 256 MiB
NAND: 128 MiB
MMC: FSL_SDHC: 0, FSL_SDHC: 1
cannot support the NAND, missing necessary info

--------------------------

//On boot after loading images via manufacturing tools
I2C: ready
DRAM: 256 MiB
NAND: 128 MiB
MMC: FSL_SDHC: 0, FSL_SDHC: 1
MXS NAND: DMA read error
NAND read from offset c80000 failed -110
*** Warning - readenv() failed, using default environment

NAND read: device 0 offset 0xd00000, size 0x0
0 bytes read: OK
Display: TFT43AB (240x320)
Video: 240x320x16

NAND read: device 0 offset 0xd00000, size 0x0
0 bytes read: OK
Error: Bad gzipped data
Error: no valid bmp or bmp.gz image at 80800000
In: serial
Out: serial
Err: serial
Net: FEC1
Error: FEC1 address not set.

Normal Boot
Hit any key to stop autoboot: 0

NAND read: device 0 offset 0x400000, size 0x800000
MXS NAND: Error sending command
MXS NAND: Error sending command
MXS NAND: DMA read error
NAND read from offset 400000 failed -110
0 bytes read: ERROR

NAND read: device 0 offset 0xc00000, size 0x100000
MXS NAND: Error sending command
MXS NAND: Error sending command
MXS NAND: DMA read error
NAND read from offset c00000 failed -110
0 bytes read: ERROR
Bad Linux ARM zImage magic!
=>

-------------------------

Also when working with the manufacturing tools via command line and USB - the board always shows up as a HID, but after jumping to the loaded image it doesn't always then show up as a mass storage device.

I get the following from dmesg for a successful load:

[30664.670765] usb 2-1.2: new high-speed USB device number 31 using ehci-pci
[30664.780675] usb 2-1.2: New USB device found, idVendor=15a2, idProduct=007d
[30664.780684] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[30664.780689] usb 2-1.2: Product: SE Blank 6UL
[30664.780695] usb 2-1.2: Manufacturer: Freescale SemiConductor Inc
[30664.783347] hid-generic 0003:15A2:007D.001F: hiddev1,hidraw3: USB HID v1.10 Device [Freescale SemiConductor Inc SE Blank 6UL ] on usb-0000:00:1d.0-1.2/input0
[30694.911322] usb 2-1.2: USB disconnect, device number 31
[30703.837465] usb 2-1.2: new high-speed USB device number 32 using ehci-pci
[30703.948007] usb 2-1.2: New USB device found, idVendor=066f, idProduct=37ff
[30703.948016] usb 2-1.2: New USB device strings: Mfr=3, Product=4, SerialNumber=5
[30703.948021] usb 2-1.2: Product: Mass Storage Gadget
[30703.948026] usb 2-1.2: Manufacturer: Linux 4.1.15+ with 2184000.usb
[30703.955704] usb-storage 2-1.2:1.0: USB Mass Storage device detected
[30703.956062] scsi host6: usb-storage 2-1.2:1.0
[30704.986801] scsi 6:0:0:0: Direct-Access Linux File-Stor Gadget 0401 PQ: 0 ANSI: 2
[30704.987755] sd 6:0:0:0: Attached scsi generic sg2 type 0
[30704.988232] sd 6:0:0:0: [sdb] 2048 512-byte logical blocks: (1.05 MB/1.00 MiB)
[30704.988965] sd 6:0:0:0: [sdb] Write Protect is on
[30704.988971] sd 6:0:0:0: [sdb] Mode Sense: 0f 00 80 00
[30704.989999] sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[30704.993403] sdb:
[30704.998412] sd 6:0:0:0: [sdb] Attached SCSI removable disk
[30876.413920] usb 2-1.2: USB disconnect, device number 32

But often it stops after this:

9839.843405] usb 2-1.2: new high-speed USB device number 30 using ehci-pci
[29839.953274] usb 2-1.2: New USB device found, idVendor=15a2, idProduct=007d
[29839.953283] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[29839.953289] usb 2-1.2: Product: SE Blank 6UL
[29839.953294] usb 2-1.2: Manufacturer: Freescale SemiConductor Inc
[29839.956259] hid-generic 0003:15A2:007D.001E: hiddev1,hidraw3: USB HID v1.10 Device [Freescale SemiConductor Inc SE Blank 6UL ] on usb-0000:00:1d.0-1.2/input0
[29879.557354] usb 2-1.2: USB disconnect, device number 30

Thanks again for your advice.

Best regards

Grant

[igorpadykov]

Hi Grant

in general this also may be caused by signal integrity issues, so one can try
to tweak drive strength of NAND signals using DSE field register
IOMUXC_SW_PAD_CTL_PAD_X_Y. Also problem with USB detection
may point to hardware issues, one can check clocks and power supplies
requirements using Hardware Development Guide for the i.MX6 UltraLite Applications Processor
https://www.nxp.com/docs/en/user-guide/IMX6ULHDG.pdf

Best regards
igor

[igorpadykov]

Hi Grant

right, commercial and industrial versions should behave identically,

also for new chip recommended to run ddr test and update image with new

ddr calibration coefficients.

https://community.nxp.com/docs/DOC-105652 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------