Why does printing cause hard faults?

[kackle123]

I am using a modified MCUxpresso OTA example (with FreeRTOS and Redlib) on a Kinetis K64.  If I unplug the Ethernet cable for a time and then reconnect it, there's a brief flurry of messages out of the UART as the program tries to recover itself.  It is after this that a hard fault occurs. 

Everything looks fine in the debugger tabs (no heap/task stack overflows), but I see that an Mbed TLS C structure ("ssl") gets corrupted, where UART text is written over one of the structure's members.  It seems this is being done by Redlib's _vsnprintf(). 

Since I don't have the source code for Redlib, how can I tell WHY this is happening?  That is, is this a malloc issue or a static buffer issue?  I know printfs are generally best avoided in firmware, but the example already uses many of them, and I want to know exactly the problem so I can make informed decisions.

[Celeste_Liu]

Hello @kackle123 ,

Thanks for your post. I found a post related to Redlib. You may refer to it to see if it's helpful for solving your problem.

Where can I post comments/suggestions etc. regarding RedLib? - NXP Community

Besides, I suggest that you try increasing the size of the heap/task stack.

Redlib is not open-source. If necessary, you can consider switching to use Newlib/NewLibNano. The specific switching method can be seen in the figure below.

To further determine whether it is a malloc issue or a static buffer issue, you can try to track the memory usage, analyze the changes of memory data before and after the function call, and other methods to try to locate the problem.

Hope it can help you.

BRs,

Celeste

[kackle123]

@Celeste_Liu

Thank you for your reply.  I will consider these suggestions. 

I see #defines in Redlib's stdio.h (like _SYS_OPEN).   I assume there is no way to recompile the library to accept changes to these #defines?

[Celeste_Liu]

Hello @kackle123 ,

The stdio.h file can be modified. Please note that after making the changes, you should close the stdio.h file and select "Save". Subsequently, the project requires a "clean" operation before being recompiled; otherwise, the modifications will not take effect. Additionally, we recommend modifying stdio.h with caution. You can back up the file or add comments to assist with restoration in case of need.

Hope it can help you.

BRs,

Celeste

--------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the "ACCEPT AS SOLUTION" button. Thank you!
--------------------------------------------------------------------------------------------------------

[kackle123]

My guesses/changes to stdio.h didn't help.

But I have since seen the hard fault occur where a printf didn't seem to be the cause.  The same spot in memory (the C structure) got corrupted though.

What I noticed afterward were anomalies in the Heap Usage (FreeRTOS) debugger tab.  The MCUXpresso_IDE_FreeRTOS_Debug_Guide.pdf manual was unclear: Under "Details", every other line shows "Allocated".  I think this allocated size is some sort of linked-list that tracks all of the blocks used by the allocation that's mentioned on the NEXT line under "Details".  If this is correct, is it normal for this Allocated size to be 40 kB when the following Task Stack line is only using 25 kB? 

[Celeste_Liu]

Hello kackle,

Based on the situation you described, the reason related to "printf" has been ruled out for the hardfault, but the same C structure in the memory has been corrupted. I suggest that you carefully check the parts of the code that operate on this memory area to see if there are situations like pointer out-of-bounds or array out-of-bounds writing. You can conduct further debugging to locate where the problem lies.

Regarding the issue that the "Allocated" size in the heap usage doesn't match the size of the task stack usage, your understanding of what "Allocated" means is somewhat reasonable. Under the memory management mechanism of FreeRTOS, it indeed records the size of the entire allocated memory block, including the part actually used by the task and the management overhead. It's possible that 40 kB is allocated while the task stack only uses 25 kB. On the one hand, there's the extra overhead of memory management, and on the other hand, it might be affected by memory alignment factors. However, it's necessary to pay attention to whether there's a memory leak. You can observe the changes in heap usage over a long period of time or use a memory analysis tool to check it out.

If necessary, you can send us your project for us to have a look. You can create a private case and tell us the detailed steps to reproduce the problem.

You may submit a case and describe your problem or what you want to know, and engineer will reply and provide related resource by email.

Please follow up below steps to submit a case:

1. Go to SUPPORTHOME, click on "Submit a Ticket" in the middle of this page. http://www.nxp.com/support/support:SUPPORTHOME

2. Login, Or Register with your business email

Hope it can help you.

BRs,

Celeste

[kackle123]

Thank you for the reply.  I can reliably cause a crash within 5 minutes; as I mentioned, I only have to remove then replace the Ethernet cable.  Before I submit a support ticket, maybe it's best that I understand exactly what the Heap Usage (FreeRTOS) debugger tab is showing.  Please tell me whether any of the following statements are incorrect.

There is the FreeRTOS heap (mine is using heap type 4), its size is defined in FreeRTOSConfig.h.  Calls to xTaskCreate() allocate their initial (function parameter) memory from this heap.  Then, a task's C code can later call pvPortMalloc() to request more memory blocks, which also come from this heap.

The memory parameter in a xTaskCreate() call is recorded as "(Task Stack)" in the Heap Usage (FreeRTOS) debugger tab.

The Task Control Block memory for each task is recorded as "(Task TCB)" in the Heap Usage (FreeRTOS) debugger tab. 

Does an "Allocated" line show the extra allocations for the line description that's ABOVE it or BELOW it?

[Celeste_Liu]

Hello @kackle123 ,

Hope you are doing well. There is no problem with the following statement, it is correct.

"There is the FreeRTOS heap (mine is using heap type 4), its size is defined in FreeRTOSConfig.h. Calls to xTaskCreate() allocate their initial (function parameter) memory from this heap. Then, a task's C code can later call pvPortMalloc() to request more memory blocks, which also come from this heap.

The memory parameter in a xTaskCreate() call is recorded as "(Task Stack)" in the Heap Usage (FreeRTOS) debugger tab.

The Task Control Block memory for each task is recorded as "(Task TCB)" in the Heap Usage (FreeRTOS) debugger tab.  "

As for the last question, " Does an "Allocated" line show the extra allocations for the line description that's ABOVE it or BELOW it?"  I can't quite understand this question. Could you please explain it further?

According to my understanding, the "allocated" row here shows the memory used for byte alignment.

Hope it can help you.

BRs,

Celeste

[kackle123]

I don't think the "Allocated" lines show byte alignment memory use because I have seen them grow significantly in size.  For example, see the attached screenshot. 

Do these Allocated lines include successful calls to pvPortMalloc(); can you confirm with someone that this is what that debugger tab is supposed to show?  Thank you.

[Celeste_Liu]

Sorry, I was indeed wrong before. "Allocated" does not represent the memory used for byte alignment.

The Heap Usage lists the FreeRTOS heap status with all the memory blocks allocated in it. The "allocated" lines represent the total amount of memory allocated through the heap. It reflects the memory usage of the FreeRTOS heap area, including the memory allocated by pvPortMalloc. The memory occupied by both tasks and other resources will be displayed here. It's just that the memory occupied by tasks is marked separately.

[kackle123 · Accepted solution]

Thank you for your reply.  The FreeRTOS OTA example I modified uses many print statements. I have avoided both mallocs and prints in my firmware career until this, my first FreeRTOS project. I suspect my non-thread-safe print library (NXP’s RedLib) is causing the issue. Removing the prints from the project fixes the problem, for example. So I’m going to focus my efforts there.

[Celeste_Liu]

I'm glad to hear that you've found the direction for further efforts. Any new issues, welcome to create the new post.