"PRINTF" SRAM Overhead/Stack Overflow

[myke_predko]

I'm running my FreeRTOS (with USB CDC) development code on a FRDM-K22F and I've just started seeing something strange.  To monitor operations, I periodically "PRINTF" (using the SDK library code) to put out a message.  All works well when I run with MCUXpresso debug active.  

If I stop the debugger (and optionally remove the USB Cable from the debug port of the Freedom board) which allows the code in the Freedom board continues executing until it encounters a "PRINTF" statement at which point it stops and indicates a Stack Overflow issue.  

The "PRINTF" statements are typically only built into the code when I specify the pre-processor symbol  SDK_DEBUGCONSOLE=0 and all "PRINTF" instances are surrounded by "#if" statements like:

#if (2 != SDK_DEBUGCONSOLE)
  PRINTF("Done");
#endif

When I remove the "PRINTF" statements by setting SDK_DEBUGCONSOLE=2 the application runs fine, no stack overflows detected regardless of whether or not it is connected to the development PC and whether or not debug is active.  

So, I believe the problem is with the PRINTF statements.  Now, I've increased the "configTOTAL_HEAP_SIZE" as well as the stack size for the first task that executes a "PRINT" statement but no joy.  25k is the total stacks size used by all the tasks and the total system heap is 36k. 

I haven't checked running the code without MCUXpresso Debug active for a week or so, during which I've added a number of tasks and queues and a mutex along with increasing the total number of queues in FreeRTOSConfig.h.   - but, as indicated, when MCUXpresso Debug is active, no issues or overflow detected/indicated.  Along with that no task's stack is close to it's threshold.  

Rather than pouding out different ideas, I'd thought I'd ask if anybody has any thoughts as to where I should look to understand this issue.  

Thanx!

[ErichStyger]

I have it here (from the FRDM-K22F ADC interrupt example):

[myke_predko]

@ErichStyger 

Regarding where is the "__assertion_failed" method.  

I can find it in "fsl_assert.c" in the "utilities" folder of the "frdmk22f_adc16_interrupt" project but...

I can't find the explicit location for the source for "__assertion_failed" despite spending a lot of time grepping it on my system.  I would expect to find it in a .c or .cpp file but no joy.  I can find it referenced in multiple files, but no explicit definition for it (or even the assert macro, although when I hover over it, I can see the macro expansion which is the same as the frdmk22f_adc16_interrupt example).  

I'm going to assume that the assert is what I can see in other projects, but as discussed elsewhere, I don't think assert is the issue.  

myke

[ErichStyger]

Hi @myke_predko 

To close on that one (even not related):

Check the .map file:

[myke_predko]

@ErichStyger 

In reply to:

To close on that one (even not related):

Check the .map file:

That's exactly what I did but the path given in the .map file seems to be truncated. 

Now, when I copied and pasted to show you what I got, the full path:

 .text.__assertion_failed
0x000000000001d36c 0x4c /usr/local/mcuxpressoide-11.3.1_5262/ide/plugins/com.nxp.mcuxpresso.tools.linux_11.3.0.202011031536/tools/bin/../lib/gcc/arm-none-eabi/9.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/hard/libcr_semihost_nf.a(_cr_assert.o)
0x000000000001d36c __assertion_failed

I'll look into this.  

myke

[ErichStyger]

I think I see the reason: in that example I'm using the SDK is overwriting the library definition of it, while on your side you are using the version from the library.

[ErichStyger]

Hi @myke_predko ,

ok, just checked an example on my side (FRDM-K22F adc16 interrupt example from the SDK):

I get that huge call stack because of the assert and the associated printf().

Could it be that in your case the assert gets triggered? but only in the case it triggers the printf() would be executed. I would disable that assert to give it a try. Or replace it with a check turning on a LED if it fails,then you know if it is really that assert.

I hope this helps,

Erich

[myke_predko]

Hi @ErichStyger 

So, other than the ADC value being stored in an array, our code is basically identical  

I expanded the Call Graph and have the same results as you do.  

When I check the assert, ChannelGroup is zero (ADC0) and FSL_FEATURE_ADC16_CONVERSION_CONTROL_COUNT is 2, so no issues with assert not being true.  

myke

[ErichStyger]

Hi @myke_predko ,

I would comment out that assert and give it a try if it makes a difference.

Erich

[myke_predko]

Hi @ErichStyger 

I commented out the assert and tried it.  

No differences in operation.

[ErichStyger]

Hi @myke_predko ,

I would set the NDEBUG define in the project settings to disable all these asserts to see if it makes any difference. Your assert is very concerning to me: in case of a triggering it it most likely will immediately overflow any stack unless you have many KBytes available for it: doing File I/O is huge (both from stack usage and code size).

Just to be sure: you are running the 'debug' and not the 'release' version on the target, right?

Erich

[myke_predko]

@ErichStyger 

Yes.  I am just building a "Debug" image, not "Release".  

I just specified "NDEBUG" in my build and the Call Graph Depth for the ADC0_IRQHandler changed to:

BUT, the behaviour hasn't changed.  

This feels like to me that the problem is somewhere else.  Thoughts as to where it could be?  

Thanx for all your time on this.  It's appreciated.

myke

[ErichStyger]

Yes, maybe back to field 0.

My understanding is this:

- here is PRINTF in your code

- if you run that code with debugger attached it works fine

- if that code runs without debugger it fails with a stack overflow at that printf statement

- you are using semihosting for the printf

Now there are two things:

- printf and its other family members are known to use a lot of stack space and cause all kind of (e.g. reentrancy) problems. Thus I don't use them

- if using semihosting, this throws a debug exception which is handled by the debugger (if attached). If there is no debug session going on it throws a hard fault. So the standard behaviour is 'printf does not work without debugger attached). To overcome this there shall be a special hard fault handler in your project (there is an option in the SDK project generation for it): look for that semihost_hardfault.c

So it would be useful how your stack overflow is detected/triggered. FreeRTOS has two different methods for this: Method 1 just compares the PSP at task context switch time while method 2 checks the pattern on the stack. If it is not a true stack overflow, something else (dangling pointer) might have written something to your stack which is not an overflow but more of a stack corruption. I would check if this is the case. And if I would set a watchpoint on that address to find out who is writing to that address.

I hope this helps,

Erich

[myke_predko]

Hi @ErichStyger 

I'm going to answer your post in two parts.  

The first is on the MSP reset.  In your screenshots, it looks like the first MSP set up is done in "prvPortStartFirstTask" which is called by the "xPortStartScheduler" method which was called by "vTaskStartScheduler".  

Looking through the project source, other than the setup in "prvPortStartFirstTask", MSP isn't accessed except in the CMSIS library code.  

I made the change to the code that you recommended in your reply:

With no change in the operating behaviour.  

Next, I tried taking out the MSP reset in "prvPortStartFirstTask" all together (as discssed in your article):

With again no change in behaviour.  

So, I'm restoring port.c (where "prvPortStartFirstTask" and the MSP setup/reset occurs) on the assumption that this isn't the issue.  

[ErichStyger]

Hi @myke_predko ,

ok, I think it it was at least worth trying.

Erich

[myke_predko]

Hi @ErichStyger 

No worries.  I'm always game to learn sometthing new as well as try different things.  

myke