8 EVKs bricked - unable to connect to flash (DAP, JLInk, MCU boot utility)

[bp1979]

Dear NXP,

I need support badly. After 2 years of development I bricked 2 EVK1021 and 2 EVK1024 EVKs. During development (while debugging) they suddenly seem to die. I never understood why. I did open a forum post earlier, but ended up just throwing them away and moving on to new EVKs / custom boards.

Yesterday 4 EVKs ended up in the same situation. Still possible to boot a RAM app in serial download, in no way possible to erase flash.

All I did, repeated on 4 EVKs, is upload my software. The program breaks in the first line of main program. When I then resume the app, I can see that the debugger becomes unresponsive, and after resetting it, it's dead. Uploading software to EVK or custom board is something I do constantly, without problems. 

Steps taken to fix EVK:

- boot switches to serial download (SBMR1 = 0x0, SBMR2 = 0x1000001).

- run blinky as RAM app (just to test) = OK

- attempt to mass erase = fails (from MCUxpresso using DAP, from JLinkExe using JLink Plus probe)

I am trying to connect using the NXP MCU Boot Utility, again, without any success. Powered as shown below

I can select the imx rt 1024 from the MCU device dropdown successfully, (so it is able to find vendor ID and product ID from human interface devices). But when I try to connect, after a couple of seconds I can see the human interface device disappear in the device manager, and this error appears:

As you can see, it did show some information about the device status. But it doesn't show any more detailed information WHY it lost the connection.

When I try to erase flash via JLink, it gives the following error

J-Link>erase
No address range specified, 'Erase Chip' will be executed
'erase': Performing implicit reset & halt of MCU.
ResetTarget() start
Core did not halt on flash image verification. Assuming faulty flash settings.
Halting target manually.
ResetTarget() end - Took 317ms
AfterResetTarget() start
MPU was enabled and is now disabled.
AfterResetTarget() end - Took 1.02ms
Erasing device...

****** Error: Timeout while erasing chip, RAMCode did not respond in time (PC = 0x20000488, XPSR = 0x21000000, SP = 0x20000A48)!
Failed to erase chip.
Failed to execute RAMCode for chip erase!
J-Link: Flash download: Total time needed: 20.126s (Prepare: 0.059s, Compare: 0.000s, Erase: 20.022s, Program: 0.000s, Verify: 0.000s, Restore: 0.044s)
ERROR: Erase returned with error code -5.
J-Link>

The questions I really need answers to:

1) Are there any other things I can try?

2) Can you explain what can be possible root causes that cause this? I literally unboxed the EVK, connected it, and bricked it by just uploading software. My last board lived for 30 seconds.....

3) Is it possible to send the boards to NXP for further analysis, it's getting rather costly to constantly order new EVKs and throw them away when "something" happens. We want to know WHY, or HOW to prevent this from happening.

[kerryzhou]

Hi @bp1979 ,

   I also simply talk your situation with our internal expert.

  He gives a testing suggestion: When in the serial download mode, please don't just use the reset button, switch the mode, you need to do the power off and power on again. Whether you still have the connection issues or not?

  If issue still happens, please share me your testing app files, let me reproduce it, if you create internal case, you also can write assign to kerry zhou, then I can continue to support you in your private case.

Best Regards,

kerry

[kerryzhou · Accepted solution]

Hi @bp1979 ,

  Please don't unbox any new board before find the root issues.

  Answer your questions at first:

1) Are there any other things I can try?

=>Answer: Seems the MCUBootutility still can find the interface, but when do the connect, it is lost, right?

Please use this link:

https://community.nxp.com/t5/i-MX-RT-Knowledge-Base/RT-board-recovery-for-debugger-connect-issues/ta...

Try the SPT tool on your side.

BTW, find another USB cable, one connect to the debugger interface, another connect to the USB interface. J1 connect 5-6.

Then try it again.

2) Can you explain what can be possible root causes that cause this? I literally unboxed the EVK, connected it, and bricked it by just uploading software. My last board lived for 30 seconds.....

=>Answer: if just can't connect, it may related to the app, which has abnormal situation, the core can't find the correct place to enter, so it needs to enter the serial download mode to recover.

I don't think the board is really bricked, so, I need to reproduce your issues on my MIMXRT1020-EVK or MIMXRT1024-EVK. You can send your project to me which can reproduce the issues.

I will help you do the testing.

3) Is it possible to send the boards to NXP for further analysis, it's getting rather costly to constantly order new EVKs and throw them away when "something" happens. We want to know WHY, or HOW to prevent this from happening.

=>Kerry:Let's check it at first, if still have issues, you can go this channel to return:

http://www.nxp.com/support/sales-and-support/buy-direct/returns-and-warranty-information/dev-tool-wa...

Wish it helps you!

Best Regards,

Kerry

[bp1979]

Hi @kerryzhou 

Thank you very much for the fast and thorough reply.

1) Are there any other things I can try?

=>Kerry: Seems the MCUBootutility still can find the interface, but when do the connect, it is lost, right?

=> Bas: Yes, that's correct. It behaves exactly the same with SPT, I made a screen capture (2023-05-16 13-49-24.mp4) and attached it to this topic.

2) Can you explain what can be possible root causes that cause this?

=>Kerry: if just can't connect, it may related to the app, which has abnormal situation, the core can't find the correct place to enter, so it needs to enter the serial download mode to recover.

=>Bas: Not sure if I understand this well enough. I would expect that I can still upload bin files with the CPU in serial download, or at the very least, I would expect that I should be able to mass erase from serial download.

3) Is it possible to send the boards to NXP for further analysis, it's getting rather costly to constantly order new EVKs and throw them away when "something" happens. We want to know WHY, or HOW to prevent this from happening.

=>Kerry:Let's check it at first, if still have issues, you can go this channel to return:

=>Bas: thanks for the link. Happy to hear that the support goes beyond limits. I will wait for your signal before I send anything.

4) Share project

=> Kerry: I don't think the board is really bricked, so, I need to reproduce your issues on my MIMXRT1020-EVK or MIMXRT1024-EVK. You can send your project to me which can reproduce the issues.

=> Bas: That would be really good news, otherwise I just burned 400 euro in a day :-). Glad there is till hope. Sharing the project is something I would have to ask permission for. But even when I could, the question is how much effort you are willing to put in this. I am developing inside a docker container, on ubuntu, using vscode with cortex-debug addon. I could also share the bin files that brick the boards but I don't know if the bin file alone is in any way useful.

5) Try the SPT tool

Please check the video to get an idea how the board behaves. I also checked the logs in c:\user\bas\secure_provisioning, which indicated that some batch script failed. I ran the script from command line piping all std out std err to log. I have no idea if it's of any use for you, but here goes anyway:

### Check presence of FlashLoader ###
FlashLoader is not running yet, download and run it
Press any key to continue . . . 
### Check communication with target bootloader ###
sdphost -u 0x1FC9,0x0130 -j -- error-status
{
   "command": "error-status",
   "response": [
      4042322160
   ],
   "status": {
      "description": "1450735702 (0x56787856) Hab Is Disabled (Unlocked)",
      "value": 1450735702
   }
}
sdphost succeeded, HAB disabled
### Build flashloader as unsigned bootable image ###
Flashloader build skipped, because result already exists: "C:\Users\Bas\secure_provisioning\bootable_images\unsigned_MIMXRT1024_flashloader.bin"
### Write FlashLoader ###
sdphost -u 0x1FC9,0x0130 -j -- write-file 0x20209C00 "C:\Users\Bas\secure_provisioning\bootable_images\unsigned_MIMXRT1024_flashloader.bin"
Status (HAB mode) = 1450735702 (0x56787856) Hab Is Disabled (Unlocked).
Response status = 2290649224 (0x88888888) Write File Success.
sdphost succeeded, HAB disabled
### Start FlashLoader ###
sdphost -u 0x1FC9,0x0130 -j -- jump-address 0x20209C00
Status (HAB mode) = 1450735702 (0x56787856) Hab Is Disabled (Unlocked).
sdphost succeeded, HAB disabled
### Waiting FlashLoader to be initialized for 3 seconds ###
### Timeout wait value can be adjusted from Preferences ###
### Check presence of FlashLoader ###
blhost -t 2000 -u 0x15A2,0x0073 -j -- get-property 1 0
blhost failed

I also tried to mass erase the flash from MCUxpresso.

[kerryzhou]

Hi @bp1979 ,

   Thanks for your updated information.

    I also checked your video, it indicate the internal flash meet some connection issues.

    Please don't worry, about your situation, I will try my best to help you, as you already meet 8 EVKs with the same issues, I want to reproduce it on my side, if I also reproduce it, I can talk about your issues internally with our NXP expert.

  BTW, do you enable any secure function or not?

   About the issue app, you can provide the .bin or the .srec file which you want to burn it with MCUBootutility or the SPT tool, just let me reproduce it.

  To your app, if you don't want to share it on the public side, you also can create the private case, then you can send me the email with your app bin file.

This is the private case create flow:

1. Open below SUPPORT site, click blue "Go to Tickets" in the middle.
http://www.nxp.com/support/support:SUPPORTHOME

2.Then you will be requested to Login, if you have no an account, please first Register with your business email.

3.After login, please "Create New Cases" button in the middle, then you can submit your question.

Best Regards,

Kerry

[bp1979]

Dear @kerryzhou 

I created the case as you suggested.

00538210

I hope I did everything correct and you can find it back, and you have enough information. I truly hope its going to reproduce, but something tells me we are not that lucky (I have a gut feeling it only occurs when in the combination or launching debugger with (too many?) breakpoints enabled).

Hence, would it make sense to already send one are two (or all... :)) "bricked" boards your way? Either you can reproduce the issue, and you don't need them, or, you can't reproduce it, and then we probably can save some time waiting for the board(s) to arrive.

You also shared a suggestion where you ask not to use the reset button when switching boot modes. I never do. I always power off and power on when switching boot modes.

>> BTW, do you enable any secure function or not?

No we don't use any secure features, or at least, not that we know of :).

[bp1979]

OMG!!! Yesterday I booted a windows 10 VM and ran SPT without much success as shown in the video I shared earlier.

I thought to give it a try from Ubuntu after I found out that there is also a native linux deb installer. IT WORKED!!!!

initialize flash

erase whole flash

I already ran blinky from flash again. I haven't tried on the other EVKs yet, but this looks promising.

UPDATE:

Just a FYI: Tried again from Windows VM. It has full access to USB peripheral on host (native ubuntu). When running SPT it can connect, but configuring flash fails as shown earlier. This might be related to some issue with running in a VM (virtualbox). Or there is something going wrong with the windows version of SPT.

For me the good news is, is that I can revive my boards using SPT on native ubuntu with the deb installer.

I am still in the progress of figuring out any clue of what is "bricking" my EVK with the current binary file I am uploading to flash. That issue persists.

UPDATE:

The program does break in main(). When I step, the debugger (GDB, controller via JLink in my case) fails and the EVK is "bricked".

As a first test I added an endless while

int main(void)

{

while (1)

{

__asm__ volatile ("nop");

}

BOARD_ConfigMPU();

BOARD_InitBootPins();

BOARD_InitBootClocks();

BOARD_InitBootPeripherals();

To my surprise, it breaks once on the nop. When I step (which should should just execute the nop again) the GDB no longer responds, and the EVK is bricked. I can get it back to life using SPT, so that's good, but no clue what causes this just yet.

If you would be able to reproduce this, and have any idea I could try I am all ears!

If I learn anything new, I will share it here too.

[kerryzhou]

Hi @bp1979 ,

   Thanks for your news sharing.

    Seems it is relation to your working platform.

    You are using the Ubuntu system, which add the VM with window10?

    Do you try the win10 system directly with the SPT or not? Maybe it is related to your PC configuration, not the EVK is really broken.

    You can find a win10 PC and try it again.

    Please also try other 7 EVKs which is bricked on your side with your working method.

   Any updated information, kindly let me know.

Best Regards,

kerry

[bp1979]

Hi @kerryzhou 

I found the root cause of "bricking" the EVKs. Or, at least, something that will have to pass as the root cause ;-). It will be a strange story, bear with me.

I am developing the ENET peripheral (integrating the SDK lwip_mqtt_freertos example in my own app, that is).

I had strange issues, where it something received an IP address from DHCP server, and in most cases it wouldn't. It looked like ENET interrupts simply wouldn't come through. So I added some statements in a freertos task which were monitoring the ENET interrupts.

This task (I just added something to an already existing monitor task, which happened to be in my main.cpp source file), just added these statements:

volatile auto e1 = ENET->EIMR;
volatile auto e2 = ENET->EIR;
__asm__ volatile ("nop");

Nothing special. Right?

Once more for clarity: I am developing on ubuntu, in vscode (instead of MCUxpresso) with the Cortex-Debug extension. When the CPU makes the jump from startup code, to main(), while I added e1 and/or e2 to my watch window, I could observe that GDB is trying to read back bytes from address

0x402D8008u

For WHATEVER reason, this fails. GDB actually shows that as well (which I completely missed, because I don't really look at GDB output).

Failed to read 2 bytes from 0x402D8008

When this happens, all goes south. vscode (/cortex-debug) no longer reacts when I try to pause/restart the debug session. The only option I have, is to stop the debug session. And that's the moment where the EVK is bricked.

So WHY this happens, no clue. WHAT exactly happens: no clue either.

But for me, the workaround is pretty easy, just remove the variables from the watch window.

My main take aways from this (for me, horrible) event:

- SPT saves the day when it comes to reviving EVKs. I am using Ubuntu, the version released for Linux (deb.bin) serves me really well. I think in the last 3 days I used it over 20 times to fix my EVK after "bricking" it.

- Something weird is going on when trying to add a watch variable to ENET. No clue why, but for now I don't need to do this often anyway, so at least I now know what is causing it. It reproduces 100%. When I add the watch again, I can execute my "fix EVK routing". Tried it a couple of times to be sure. After removing the watch, debugging works normally. I ALSO checked how MCUxpresso behaves when I add the same watch variable. It doesn't care about it at all. It works as you would expect. So it's a vscode/cortex-debug/gdb issue. Also, I use arm toolchain 12.2 instead of the one that ships with MCUxpresso. So yeah, go figure who's responsible for this special issue....

PS: For me the best news ever, my ENET peripheral now works next to the PPPos interface. So one happy camper here ;-).

Dear Kerry, thank you so much for being so insane supportive. I really (!!!) appreciated it!

PPS: I don't have a windows PC to test SPT on native windows, but I am sure you can, and I am also sure it will work just fine.

Thx again!

BR Bas

[kerryzhou]

Hi @bp1979 ,

  You are always welcome!

  So, in fact, none of your EVK is broken now.

  I am glad that you find the root issues.

  If your issue is solved, please help to mark the correct answer, to close this case. 

  If you still need my help, just kindly let me know.

Best Regards,

Kerry

[bp1979]

Done that. Thx again.

And yes, all board are working again. Or at least, the ones I had left. In the last year I bricked the first 4 which I was convinced that were broken, so I threw them in the garbage bin and bought 4 new ones :).

Note to self: no more throwing away!