MCUX 10.2.1 - Debug launch freezes at 2% for about a minute

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

MCUX 10.2.1 - Debug launch freezes at 2% for about a minute

Jump to solution
7,176 Views
dmarks_ls
Senior Contributor I

So this isn't a new problem with 10.2.1... to my recollection, this problem has existed for a few months at least.  The issue is that whenever I do a debug launch (green bug), on the bottom status bar, I see "Launching foo_bar ... Debug: (2%)" with a short green progress bar, and it just strobes away for a minute or two while the Console window is blank.  Nothing else seems to be happening in the IDE while it's at 2%.  After this lengthy delay, it finally jumps to 6%, does a build prior to launch, usually there's nothing to do for the build, and then it goes to launch GDB.  Happens with both PEMicro and SEGGER debug launches, and happens with each of two different Kinetis projects I'm currently working on.

The only time this doesn't happen is if this is the first time I've launched the debugger since starting MCUX.  In that case, clicking the green bug makes the progress go immediately to 6%, the pre-launch build happens, and then the debugger starts.  If I then terminate the debugger, and then click the green bug again, it sits at 2% for another minute or so, then goes to 6%, does the build, etc.

This is a bit of a productivity killer, having to wait a couple of minutes for no good reason each time I do a debug launch.  Any idea what's causing this pause where seemingly nothing is happening in the IDE?

(Win 10 Pro 1803 patched, MCUX 10.2.1, Kinetis K24 project)

1 Solution
6,125 Views
lpcxpresso_supp
NXP Employee
NXP Employee

This issue should be resolved with MCUXpresso IDE v11.0.1:

 

Regards,

MCUXpresso IDE Support

View solution in original post

46 Replies
2,980 Views
dmarks_ls
Senior Contributor I

So, I tried resetting the Debug perspective (yes, that's the perspective I use when debugging).  I then started a debug session, it went right into debugger launch, SEGGER programming window came up, chugged, then went away, and the Debug tab then showed the GDB hierarchy.  But also, tabs immediately appeared in the lower tab group (console, etc) for:

  • Global Variables
  • Faults
  • Memory

And in the top-right tab group (breakpoints, etc) for:

  • Registers
  • Modules

So something is actively stuffing those modules into the Debug perspective when a debug session is launched.  I tried what you suggested... I reset the Debug perspective, then saved the perspective as "Debug (Custom)".  Then I went into Preferences / Run/Debug / Perspectives, and I set "C/C++ (NXP) MCU Application", "GDB PEMicro Interface Debugging", and "GDB SEGGER Interface Debugging" to use "Debug (Custom)" instead of "Debug" when launching.  And hey, how about that, MCUX didn't add those extra tabs to my custom perspective the next time I launched.

So that's the issue mitigated, for me at least.  Now we just need to figure out why launching the Global Variables tab is so damn slow with C++ projects, to make sure it doesn't affect any other users.

0 Kudos
2,709 Views
lpcxpresso_supp
NXP Employee
NXP Employee

We are unfortunately still failing to replicate the issue you are seeing. We'll continue to investigate, but we have a few ideas for further things you could try, to see if they have any effect ...

 

  1. Before starting you debug session, try disabling all breakpoints (there is a "Skip All breakpoints" button on the main IDE icon bar that can be used to do this).
  2. Before starting your debug session, try closing all non-essential views - for instance Expressions, Global Variables, Variables, Outline
  3. Rename your existing launch configuration (for example, change the file extension to .oldlaunch), and then use the Quickstart Panel's Debug button to do a probe discovery / generation of a new launch configuration.

 

Regards,

MCUXpresso IDE Support

2,654 Views
dmarks_ls
Senior Contributor I

Well done, I think you've isolated it.  It's the Global Variables view that's mucking things up.

I did all three of those items above (breakpoints window was already empty, closed all unneeded views, and recreated my launch file) and launched a debug... went straight into debug, great.  Terminated, then immediately restarted... and it sat at 2% again.  While it was sitting there, I saw that the "Global Variables" view had re-inserted itself into the lower tab group, so I clicked to look at it... the cursor went into a spin-wait and the GV window contents didn't draw until the 2% delay was done.  Hmm.  Terminate, kill the GV tab, relaunch... no delay.  BUT, the GV tab re-inserted itself into the lower tab group again.  So, kill the GV tab, relaunch, no delay.  Terminate, kill the GV tab again, relaunch... no delay.  Terminate, leave the GV tab in place, relaunch... long delay.  So it's definitely the Global Variables tab creating the delay.

On some level, I guess I can understand why the GV view would take a long time to initialize.  My C++ program is currently 550 KB in size, so there's a lot of C++ symbols to sift through.  We haven't done any large C-only projects recently, so I really don't have a comparison to see what a ~500 KB C program would do in the same situation.

So... how do we solve or mitigate this?  It's annoying that I now have to remember to kill the GV tab every time I debug, or suffer the time penalty on my next launch if I forget to do so, since the GV tab re-opens itself during every debug launch.  One mitigation would be to suppress the re-launching of the Global Variables view each time; not sure if that's selectable in Preferences.  The fix, if it's possible, would be to somehow make launching the Global Variables view much quicker, without spending 1-2 minutes invoking the C++ name demangler.  Ideas?

0 Kudos
2,635 Views
lpcxpresso_supp
NXP Employee
NXP Employee

@David - The thing about "arm-none-eabi-c++filt.exe" (the C++ name demangling tool) is interesting. We'll investigate.  But this does make it sounds like your issue may actually be different to Harald's.

One other thing to try in the meantime might be to turn off the IDE's automated "build before debug" functionality. And then build manually yourself before you then click Debug.  The easiest way to do this is untick the "Build (if required) before launching" option in:

  • Window -> Preferences -> Run/Debug -> Launching

Does this make any difference to the behaviour that you see?

Regards,

MCUXpresso IDE Support

0 Kudos
2,523 Views
dmarks_ls
Senior Contributor I

OK, tried that, and the only thing that changed is that the IDE bypasses the build and goes straight to debug launch.  The delay at 2% is still there, and "arm-none-eabi-c++filt.exe" is still being invoked for the duration of that delay.  The console is empty during the 2% delay, and only fills when the build starts or the debug launch begins.  And can again confirm, it doesn't happen on the first debug launch after starting the IDE, only on the second and subsequent launches.

0 Kudos
2,504 Views
lpcxpresso_supp
NXP Employee
NXP Employee

I'm glad to hear that things are now at least working better for you.

When you do see the problem again, then if you are able to supply any of the information we asked about in our previous post (https://community.nxp.com/thread/488000#comment-1149907), or else try to narrow down what other things in your system **might** be contributing, then that might gives us some clues as to what exactly is happening in your particular setup.

Regards,

MCUXpresso IDE Support

0 Kudos
2,503 Views
dmarks_ls
Senior Contributor I

Hi there, OP here... I've been busy in development, but I have some time to come up for air and see if maybe we can figure this out.

My test setup is Windows 10 1803, with a SEGGER J-Link Pro running the latest firmware as of J-Link v6.46, and MCUX 10.3.1.  Target is an RT1050 EVKB (yes, I have the OpenSDA debugger disconnected from the chip).  I'm running ESET Endpoint Antivirus, but for the test below, I had real-time protection disabled.

So, first up... I tried turning off debug probe detection for all probe types.  No effect... second and subsequent debug launches still take a lot of time.  Then I started snooping the Task Manager to see if there were any lingering debug processes, as you suggested.  I did find it peculiar that mcuxpressoide.exe was coming and going from the Processes / Apps listing while the debug was launching, and was absent from the list during one particular run, even though the IDE was clearly still running.

Then I switched over to the Details tab so I could get a proper alphabetical listing of all running processes.  And while the IDE was showing 2% for the debug launch, I saw one particular process appearing, disappearing, and changing process number for the entire time that the progress was stuck at 2%:

slow_debug_cpp_filt_process.png

That "arm-none-eabi-c++filt.exe" process seemingly launched dozens of times while the IDE was stuck at 2%.  Once the value ticked up to 6%, that process went away, the IDE did one final "make", and then the J-Link GDB server process launched.

So maybe I've answered the question of what is taking up time during the debug launch, and perhaps why it's only afflicting my C++ projects.  Can you help with the answer of why this process is running?  What is it actually doing?  And why does it only run on the second and subsequent debug launches after starting the IDE?

0 Kudos
2,504 Views
haraldadolph
Contributor III

Update #4:

Hi all, so after many trials and buying a JLink basic adapter, I can say that this configuration is the most stable up to now.

It's not 100% stable, but you can start and really use the debugger maybe about 50 times without interruption.

If it stops at 2% - yeah, it happens with this box, too - it helps to close MCUXpresso, and after a restart you can go on with the next 50 trials.

This is a method I can "survive" with. So, PEMicro - both debuggers,  the standard as well as the FX - seems to be the second solution at least in my case.

Thanks for the support.

0 Kudos
1,604 Views
lpcxpresso_supp
NXP Employee
NXP Employee

I'm sorry the delay in responding here - but this one is really perplexing!

So the first thing to say is that we still haven't managed to reproduce your issue, and we are also struggling to see with the current information how the problem can be directly related to C++ or FreeRTOS projects.

From the fact that turning off probe discovery for probe types that you are not using appears to solve the problem, this would indicate the something in the probe discovery process / USB system is probably causing the problem. And the type of project, and its size, should not have any effect on this at all (despite what you seem to be seeing).

Anyway,  we are really going to need your further help to try to reproduce the problem or at least narrow down where the problem might lie.

So first of all, it would be interesting to know which probe type you had to turn off in order to get rid of the 1 minute delay at 2%. As far as I can tell, the probe being used when the issue is seen is P&E. So was it turning off LinkServer or was it turning off SEGGER that solved the issue? Or did it require both to be turned off?

Another thing that would be interesting to know is if, when you are stuck at 2%, are any of the debug related processes running? So looking at Task Manager or similar can you see any of:

  • arm-none-eabi-gdb
  • crt_emu_cm_redlink
  • redlinkserv
  • pegdbserver_console
  • JLinkGDBServerCL

Can you also look to see what version of SEGGER you have "active" for your workspace, and also ensure that "Enable discovering of SEGGER J-Link IP probes" is disabled in:

  • Window -> Preferences -> MCUXpresso IDE -> J-Link Options

[Note: On Mac this is on the MCUXpresso IDE menu, rather than the Window one]

Can you also confirm what version of the P&E plugin you have installed using:

  • Help -> About MCUXpresso IDE -> Installation Details -> Installed Software

and looking for the "GNU ARM PEMicro Interface Debugging Support" line.

It would also be interesting to eliminate the potential effect of as many other USB peripherals in your system as possible. If possible unplug everything other than your debug probe, and see if you still see the problem. And if not try adding your USB peripherals back one at a time until the problem potentially reoccurs. Something else that would be interesting is, if you are using a PC with some form of dock (which are often USB based), try removing your PC from that, and see if the problem still occurs.

Then if the problem still occurs with the minimal set of USB peripherals you can get away with being connected, please let us know what the peripheral set is - this might highlight commonalities that might be contributing towards triggering this issue. One way to do this on Windows would be to post the output from running the "usbdiags" script that can be found in the \ide\binaries\Scripts folder of your IDE installation - from before you try to start a debug session, whilst it is hung at 2%, and then after you make the debug connection.

Finally, please can you also confirm the exact debug probe (and firmware version) that you are using, along with the host OS of your PC.

Regards,

MCUXpresso IDE Support

1,603 Views
haraldadolph
Contributor III

Update #3:

The latest tip appears to be an appropriate solution to the problem.

I verified this for several days now and had no hangings any longer at 2%.

To answer your question: (although already mentioned above):

the problem appears only in connection with C++, at least in my applications.

So, hopefully you'll find a final solution for the next release.

Thanks for your support although it came a bit late ...

0 Kudos
1,345 Views
lpcxpresso_supp
NXP Employee
NXP Employee

We have so far been unable to replicate any delays of this sort of length here when starting debug sessions (though we haven't been able to find any non-FX Multilinks to try with).

We will continue to try to replicate, but in the meantime one thing for you to try would be to disable the autodiscovery of probe types that you are not using . So if you are only using a P&E probe, turn off discovery of LinkServer (typically CMSIS-DAP) and SEGGER J-Link probes. You can do this in:

Window -> Preferences -> MCUXpresso IDE -> Debug Probe Discovery

[Note - on Mac, Preferences is on the "MCUXpresso IDE" menu entry, rather than Window]

You can also set these options from the probe discovery dialog displayed the first time you debug a project.

It would also be interesting to note if this issue is only ever seen with C++ projects. And only with FreeRTOS using projects.

Regards,

MCUXpresso IDE Support

0 Kudos
1,345 Views
haraldadolph
Contributor III

Update #2:

I could verify the positive behavior with the Multilink FX as described above ... and as there doesn't seem to be a quick solution for that, I bought one FX - that costs me about 500 $ - but better than killing my project. Will get it on start of next week.

Hopefully after this easter week someone from NXP will quickly take care of this item ... otherwise the FX is my last hope that it runs stable ..without this annoying effect.

0 Kudos
1,345 Views
haraldadolph
Contributor III

Hi,

next update -- the "hanging at 2%" error returns when the "Clean up Debug" button is pressed, with the "USB Multilink" connected.

I am waiting again ...

Tomorrow I'll be able to verify this behavior again ...

I share the opinion of David ... with these informations it should be very easy for NXP developers - maybe in contact with PEMicro - to solve this quickly ...

1,345 Views
haraldadolph
Contributor III

Hi again,

further testing results (without guarantee to work generally - but if they help one of you, it's worth while discussing)

Today I had the possibility to test my software on the PC of a colleague - with a MultiLink FX debug probe (I myself have a standard USB Multilink. On this PC, the debugger was startable with my project in a completely normal way, each time the debug is activated, without any waiting time.

So we tried his debug probe connected to my PC, and amazingly it worked completely normal at my PC, too. (Remark: my PC updated my colleague's probe to SW version 10.6)

Even more amazingly, after having this result, I reconnected my own probe and I had no hanging at 2% any longer.

As I mentioned, I do not know whether this is a general result, and whether this behavior withstands several reboots of the PC, but it may be a hint for you.

In fact, the problem seems not solved to me ...

0 Kudos
1,345 Views
haraldadolph
Contributor III

Hi altogether,

some additional information from my side:

If the number of CPP is increased, waiting time goes even higher. it is better to restart McuXpresso completely instead of waiting for the debugger to start on the second start.

In addition to that, there is no restart possible in the debugger - which makes things even worse.

My hardware is a K66M180, with a PE micro debugger connected. MCUXPresso is v10.3.1, Build 2233.

Considering this malfunction of the debugger, development of a relatively large and complex system is impossible.

In my opinion, David and I gave a lot of hints where to start with the solution.

So, I appreciate your assistance and a quick solution to this problem.

0 Kudos
1,345 Views
dmarks_ls
Senior Contributor I

Yeah, I've had some hit-and-miss results with debugger stability with some of my larger C++ Kinetis projects.  I've had cases where I'm doing RTOS-aware debugging, and I take a couple of steps, and the debugger just completely loses track... can't pause it, no source code context, or anything.  If I turn off RTOS-aware debugging, I then regain the ability to breakpoint and single-step, but of course I've then lost the ability to inspect the state of all my tasks.  Sometimes I'd switch between a PEmicro probe and a SEGGER probe because the SEGGER seemed more stable in certain circumstances.  Sometimes I get the feeling that we're getting exactly what we paid for in terms of the tools quality... but I really don't want to switch to something like IAR, because their (non-Eclipse) IDE sucks.  Say what you will about the debugger, but Eclipse really is an excellent environment for developing C++ code.  I just wish that NXP actually cared about ironing out these wrinkles in their tools.

0 Kudos
1,345 Views
haraldadolph
Contributor III

Hello

I encountered the same problem as David does (or did).

It comes up as soon as I change a project from C to a mixed C/C++ project.

I did this with the lwip_httpsrv_freertos project.

In fact, there were many changes necessary in order to get it running (I need this C++ capability in order to integrate code from a parallel project). As soon as you want to load the C++ linked code (with newlib, with or without semihost), you encounter this long waiting time. I saw that David also has a project containing C++ code, so that smells similar.

Is there already a solution for that item?

1,345 Views
dmarks_ls
Senior Contributor I

I haven't seen any sort of solution to this yet.  I'm currently running MCUX 10.3.1, and it still does this for both my Kinetis C++ project and my new RT1050 C++ project.  It's interesting that you also get this when running a C++ project.  With my RT1050, I started with a simple SDK project and added in my libraries; my current project builds to about 430K of code.  And yes, if I go to debug one of the simple SDK examples, the debug launch goes fairly quickly.  But if I debug my C++ project, on the second or subsequent debug launch, progress freezes at 2% for about a minute or so before carrying on.

Surely one of the NXP MCUXpresso IDE developers can run a simple test and see what's going on?  Just make a C++ project for Kinetis or RT1050, include newlib, make a few STL calls (make some vectors, etc) to get the code size up, then try debugging the project a few times.  You'll see progress freeze at 2% for a period of time.  If you're the IDE developer, surely you can easily figure out what the IDE is actually doing at this point and why it's taking a minute or more to do basically nothing?

0 Kudos
1,345 Views
eldarfiring
Contributor III

Hi,

I have the same experience, and it has been like this since the beginning of MCUXpresso as far as I remember. Really annoying. That's one of the reasons I tend to use Ozone/Segger for debugging instead. MCUXpresso debug environment has some other nice features, so I hope it will be fixed soon.

I have several C++ projects with sizes 100-600K of code. MCUXpresso 10.3.1. This problem has been observed for all my projects.

1,345 Views
BlackNight
NXP Employee
NXP Employee

Hi David,

Can you check/disable your virus scanner/firewall if this has an impact?

I have seen such a thing on some student machines in my university classes: the virus scanner (the Avast one seems to be one doing this kind of things) tries to check on an internet data base if the debugger could be a virus and delays things up to several minutes. So disabling (or better: uninstalling) such a bad scanner solved the issues.

I hope this helps,

Erich

0 Kudos
1,345 Views
dmarks_ls
Senior Contributor I

I tried disabling ESET Endpoint Antivirus for a duration of time, but it had no effect.  When the progress is sitting at 2%, if I have Task Manager open, I can see that MCUX is consuming some appreciable amount of CPU (less than 10%), and I don't see antivirus or any other process spiking during that time.  It's just frustrating to have to burn a minute or two every time I click the debugger.  (I know, I should just write bug-free code instead.)

0 Kudos