iMX28 Mfgtool File Transfer Failing

[markwilliams]

Hi all,

We have been successfully using mfgtool to program our iMX28 NAND with Windows CE 6.0 for the last few years. I have recently had to modify the UUT project source and rebuild it for use with mfgtool. Since rebuilding I am able to program eboot and the nk image to flash but the file transfers fail.

Eboot and the NK image are written with custom ioctl codes. The file writes first get a file handle using createfile but this is where I have an issue.

At the point where the files are to be transferred it seems to crash in a 'CreateFile' call with a DMA timeout. This timeout debug message is only output under an actual DMA timeout condition (I checked in case the error was something else and the debug message misleading). Using the same UCL xml file the process completes for our earlier build UUT project.

I am currently working through all BSP changes that have been made to see if anything could be preventing the NAND from writing the files but I have not found anything yet. While I search further I am raising this discussion in case anyone has suffered this issue and may be able to help.

The debug output during the mfgtool transfer is below:

Windows CE Kernel for ARM (Thumb Enabled) Built on Mar  8 2013 at 17:05:33

INFO:OALLogSetZones: dpCurSettings.ulZoneMask: 0xb

OALInitCpuHclkClock:Update CPU clock to 454MHz!

WARN: Boot arguments not found, use defaults

DeviceId................. MX280

pArgs->flags............. 0xC

pArgs->devLoc.IfcType.... 0

pArgs->devLoc.LogicalLoc. 0x80080000

pArgs->devLoc.PhysicalLoc 0x80080000

pArgs->devLoc.Pin........ 0

pArgs->ip4address........ 0

pDevice->Name............  s

pDevice->ifcType......... 0

pDevice->id.............. 0x80080000

pDevice->resource........ 0

pDevice->type............ 2

pDevice->pDriver......... 0x8049902C

WARN: OALKitlInit: KITL Disabled

reset GPMI module

enable bch module

NAND FullCode = 0x1590da98

NAND FullCode = 0x0

NumberOfChip=1

NAND_ECCConfLayout, pagesize=2048,sisize=64

INFO: Set NAND flash blocks [0x0 ~ 0x217] as reserved.

BOT_InternalInit with Drivers\Active\07

SDHC: + DllEntry, Enter

UFN_ATTACH with key = Drivers\Active\07

UFN_CONFIGURED: BOT_OpenInterface again

UTP command:Media type: NAND.

UTP command:Check if disk is ready.

Please be patient to wait...

Send storage Friendly name: NAND FLASH Storage.

Send time out: 10.

Find Store Device Name is DSK2:

Find Store Friendly Name is NAND FLASH Storage

Info:Find Store by Friendly Name NAND FLASH Storage

Get Store disk name:DSK2: by Friendly name NAND FLASH Storage.

GetStoreList:szDisplayName is DSK2: NAND FLASH Storage

GetStoreList:szDisplayName is DSK1: SD Memory Card

DSK2: is ready.

UTP command:Write image.

UTP command:Image type: EBOOT SB.

UTP command:Send data.

Whole data length to be sent: 0xa09b0.

INFO: Programming NAND flash blocks 0xa

INFO: Programming NAND flash blocks 0x10

INFO: Programming NAND flash blocks 0xb

INFO: Programming NAND flash blocks 0x11

INFO: Programming NAND flash blocks 0xc

INFO: Programming NAND flash blocks 0x12

INFO: Programming NAND flash blocks 0xd

INFO: Programming NAND flash blocks 0x13

INFO: Programming NAND flash blocks 0xe

INFO: Programming NAND flash blocks 0x14

INFO: Programming NAND flash blocks 0xf

INFO: Programming NAND flash blocks 0x15

INFO: Set NAND flash blocks [0x0 ~ 0x217] as reserved.

Recieving data: 100% finished.

UTP command:Write image.

UTP command:Image type: NK NB0.

UTP command:Send data.

Whole data length to be sent: 0x3200000.

INFO: Set NAND flash blocks [0x0 ~ 0x217] as reserved.

Recieving data: 100% finished.

UTP command:Check if folder is ready.

Please be patient to wait...

Send folder name: FlashDisk.

Send time out: 10.

FlashDisk disk is ready.

UTP command:Write file.

UTP command:File name to be sent: FlashDisk\test.txt.

CreateFile FlashDisk\test.txt file.

-DMA_Wait(ERROR_DDI_NAND_DMA_TIMEOUT)

Flash PDD Error: Function: FlashPdd_IoControl,  Line: 365-DMA_Wait(ERROR_DDI_NAND_DMA_TIMEOUT)

Flash PDD Error: Function: FlashPdd_IoControl,  Line: 365-DMA_Wait(ERROR_DDI_NAND_DMA_TIMEOUT)

On the old UUT build the debug is exactly the same except the last stage:

CreateFile FlashDisk\test.txt file.

UTP command:Send data.

Whole data length to be sent: 0x670.

Prepare to receive file data...

Writing File data: 100% is finished.

Close file.

Whole update work is finished successfully. Please power off the board.

I am also wondering if anyone could help with how to debug this project in visual studio? As this is loaded directly into RAM by mfgtool I am not sure if I can connect and step through the image in VS? At the moment I am using retailmsg to output progress.

The only other thing I can think of is that there has been a Windows Update that has broken this functionality in the UUT project.

Thanks in advance,

Mark

[markwilliams · Accepted solution]

Just to close off the above my fix was as follows:

The flash driver relies on IOCTL_DISK_GET_STORAGEID to set a global flag called 'm_LoadDone'. When this is set the NAND driver turns on/off the GPMI module clock on each access. This way the module is only clocked when needed and not when idle (assumed for power saving). The issue was caused because the Microsoft fsdmgr code used to send IOCTL_DISK_GET_STORAGEID when mounting the store and this was removed. Without this IOCTL
code the NAND driver did not perform the clock gating on and off.

In order to transfer OS and eboot the UUT image uses custom IOCTL codes in the NAND driver. These were coded to turn on/off the NAND clock. This meant that after the OS transfer the NAND clock was disabled but because m_LoadDone was not set (IOCTL_DISK_GET_STORAGEID not sent on disk mount) the clock was not turned back on for disk read/writes by the file system. Without a clock the DMA timeout would occur as the transfer never completed.

This issue was caused because the driver relied on a specific IOCTL and the decision to call this was in Microsoft code. This is not a good way to add a sync point as if Microsoft change their code in an update (as they did) then the code fails.

To fix this issue the nandbootburner.c file has been updated to comment out all instances of BSPNAND_SetClock(FALSE). The nandbootburner.c file is in two places (although only one would be compiled into the image) so both have been updated:

C:\WINCE600\PLATFORM\iMX28-EVK-PDK1_9\SRC\COMMON\NANDBOOTBURNER\NandBootBurner.c
C:\WINCE600\PLATFORM\COMMON\SRC\SOC\COMMON_FSL_V2_PDK1_9\BOOT\FMD\NANDBOOTBURNER\NandBootBurner.c

Mark

[markwilliams]

Just further to the above some may be wondering why this issue only affects mfgtool and not the normal booting Windows CE OS.

It looks like on flash driver init the clock is turned on. It is only then subsequently turned off by one of the custom ioctl codes used by mfgtool (such as those used for transferring eboot or nk images).

So if the board is booted and never sees one of the custom IOCTL codes then the nand clock is never turned off.

I can only assume the intention of turning the NAND clock on and off around each access was to save power. I.e. why clock the GPMI module if there is no access and NAND access is relatively infrequent.

Mark

[markwilliams]

Well after a painful couple of weeks I think I have tracked down the cause of this. I added some debugging to output every IOCTL call to the flash driver to study what is going on.

It turns out that the Freescale Driver Code is relying on the IOCTL_DISK_GET_STORAGEID to be sent by the Microsoft driver to set a global flag that is used to turn the NAND clock on and off each time it is accessed:

C:\WINCE600\PLATFORM\COMMON\src\soc\COMMON_FSL_V2_PDK1_9\NAND\FMDWRAPPERPDD\BIG_SECTOR_INTERLEAVE\FmdWrapperPdd.cpp

LRESULT FmdWrapperPdd::IoControl(
DWORD IoControlCode,
PBYTE pInBuf,
DWORD InBufSize,
PBYTE pOutBuf,
DWORD OutBufSize,
PDWORD pBytesReturned)
{
if (IoControlCode == IOCTL_DISK_GET_STORAGEID)
{
DEBUGMSG(TRUE, (_T("FLASH Init Step Finish\r\n")));
BSPNAND_SetClock(FALSE);
m_LoadDone = TRUE;
}

Studying the changes made to the code behind fsdmgr_lib.lib I can see that Microsoft used to send out IOCTL_DISK_GET_STORAGEID when mounting the disk which would have set up the m_LoadDone flag. In the later code they stopped doing that.

The problem with mfgtool is that one of the last IOCTL codes before the DMA timeout is IOCTL_DISK_VENDOR_END_WRITE_IMAGE which completes the reception of the OS image. The issue here is that the code behind this function turns off the NAND clock: BSPNAND_SetClock(FALSE);

When the fmd tries to write to the disk in the CreateFile call (part of transferring files using mfgtool) it tries to use IOCTL_FLASH_PDD_WRITE_PHYSICAL_SECTORS.

The problem is that this will only turn the NAND clock back on if m_LoadDone is set:

LRESULT FmdWrapperPdd::WritePhysicalSectors (
IN ULONG TransferCount,
IN FLASH_PDD_TRANSFER WriteList[],
OUT ULONG* pWriteStatus)
{
LRESULT Result = ERROR_SUCCESS;

if (m_LoadDone)
BSPNAND_SetClock(TRUE);

This then tries to access the NAND without the NAND clock on which results in the dma timeout waiting for the NAND operation to complete.

So in summary it seems that because Microsoft decided to stop sending IOCTL_DISK_GET_STORAGEID the NAND driver fails because it is relying on this IOCTL to set a global flag that controls the NAND clock!

I now need to work out the best way to fix this so need to understand the reason behind the m_LoadDone flag. If anyone is following this I would appreciate some help! I am sure like the other issues I have had to track with the iMX28 this will also affect NAND drivers across other platforms as the code is shared.

For reference, during the flash driver initialisation / mounting the store the microsoft code used to send IOCTL code 0x71C24 (DISK, 0x709, File any access, Method Buffered) which is IOCTL_DISK_GET_STORAGEID (which the freescale driver is looking for). The later code replaces that call with 0x75C08 (DISK, 0x702, File read access, Method Buffered) which is IOCTL_DISK_READ.

Really not sure why the driver code waited for IOCTL_DISK_GET_STORAGEID to be received before gating the NAND clock. Seems a bit naughty to use a microsoft IOCTL call as a sync point for something when we are not in control of what changes Microsoft may make to their drivers.

Mark

[markwilliams]

Ok so I replaced fsdmgr_lib.lib (C:\WINCE600\PUBLIC\COMMON\OAK\LIB\ARMV4I\RETAIL) with one from our old build machine prior to the 2014 FSDMGR Windows OS Update and the file transfer using mfgtool works.

So as per the post above something in fsdmgr_lib.lib has changed due to a Windows Update in 2014 and is not compatible with the UUT project. It is interesting that it does not affect running windows CE, just the transfer of files when running the UUT project through mfgtool.

The code stops in the CreateFile call so I can try and add debug messages to the fsdmgr code and rebuild fsdmgr_lib.lib. Comparing the source differences there is a reasonable amount of code change. Really not sure why this would cause DMA timeout unless the NAND driver is being blocked by the FSD code.

So work-around - use old lib file / source but then risk the issues corrected by the CE update. I will try and find the root cause by studying the driver code.

Mark

[markwilliams]

I have had to dig up this thread because it is now getting critical that we can rebuild the mfgtool UUT image to support a new NAND.

I found a similar post. This seems to hint that the 2014 WinCE update seems to have broken the ability to transfer files using mfgtool. I have studied the differences but cannot see why this would be the case.

https://translate.google.com/translate?hl=en&sl=zh-CN&u=https://social.msdn.microsoft.com/Forums/vst... 

I built our original UUT image for mfgtool to support our NAND before 2014 and this has been working fine. I am now trying to support a new NAND as the old part is obsolete but after rebuilding mfgtool (even using original NAND) I get the DMA timeout error in my original post, also backed up by another forum member, Martin above.

The OS image and Eboot update but the file transfer part uses Windows calls to CreateFile.

It seems that the error occurs in the CreateFile call in BOOL UceOpenFile(TCHAR* ptchFileName, HANDLE &hFile) in the file:

C:\WINCE600\PLATFORM\COMMON\src\soc\COMMON_FSL_V2_PDK1_9\UUT\uce_media.cpp

The code is able to find my folder FlashDisk so the drive is present and mounted but cannot write to the NAND.

I will try the suggestion from the post to revert the storage manager back to before the 2014 update and see if this fixes it.

Mark

[Bio_TICFSL]

Hi Mark,

Take a look at your nand driver code, it looks like you need to increase timeouts, in NAND_READ_PAGE_TIMEOUT(10000000)

I guess there is no way to debug this with VS.

Hope it helps

[markwilliams]

I have had a look at the timeouts. The "-DMA_Wait(ERROR_DDI_NAND_DMA_TIMEOUT)" error message is only output if the NAND write times out (in DMA_Wait() function).

The current timeouts in us are as follows (in microseconds):

#define NAND_READ_PAGE_TIMEOUT      (2000000)

#define NAND_RESET_TIMEOUT          (10000000)

#define NAND_WRITE_PAGE_TIMEOUT     (10000000)

#define NAND_ERASE_BLOCK_TIMEOUT    (20000000)

You would think 2 seconds is ample time to wait for a NAND write to complete? If I have no success using an older image I will try increasing the timeout. If it works I will add some debug to see what the time taken to read is. I would have hoped much quicker than 2 seconds per page!

Mark

[MartinHo]

Hi Mark,

I just did a quick check with my Build, and I'm having the same problem!

Unfortunately I don't have time to investigate at the moment.

best regards

Martin

[markwilliams]

Martin,

Thanks for letting me know that you are suffering the same issue. I have been through all of the changes from the raw BSP to where we are now and cannot see anything that I have done that could break the CreateFile call.

I am rebuilding our earliest build machine snapshot to see if it works back then. Our UUT image was created early in our product development when it needed changing to support our NAND. Since then we have not had to rebuild it.

We have written our own mfgtool to update the OS over USB in the field. We have found some issues with mfgtool in the verification of the image in that it appears that if the flash write fails it still returns success so you would think you have written a good image only to find potentially a random portion could be corrupt. We want to add a checksum and look to return the write status. We also need to support a new NAND.

So I will need to rebuild the mfgtool but need to understand why the NAND writes work for the OS and Eboot (direct NAND writes) but are failing via the file system.

Mark