McuXpresso v11.2.0 and v11.2.1 toolchain is broken: it is not thread safe

[v_canoz]

Hello,

After having lost 3 weeks (day and night) debugging random hardfaults in my McuXpresso project with LPC55S69 as a target, we've finally found that McuXpresso v11.2.0 and v11.2.1 toolchain is broken (at least for LPC55S69) in a multi threaded environnement such as FreeRTOS.

For example, using snprintf or just adding doubles in multiple threads at the same time will cause it to corrupt the stack.

The last working version is v11.1.1. Attached is the project which will show you the problem. I can provide more information if necessary.

To whom it may concern... I wonder how many persons have lost countless hours, since the release of v11.2 in July 2020?

@davenadler, @ErichStyger, @myke_predko this might interest you

Victor

[i_kamal]

First, a mistake in my post, i meant McuXpresso version 11.2.0 (not 12.0 - as this does not exist yet!)

So, with that out of the way:

Newlib:

#define _NEWLIB_VERSION "3.1.0"

FreeRtos:

#define tskKERNEL_VERSION_NUMBER "V10.3.0"
#define tskKERNEL_VERSION_MAJOR 10
#define tskKERNEL_VERSION_MINOR 3
#define tskKERNEL_VERSION_BUILD 0

How do I retrieve the GCC version? (sorry..) I suspect it's 9.2.1, as i see this in the path: 

"tools\arm-none-eabi\include\c++\9.2.1"

Thanks!

[davenadler]

This **bleep** forum keeps losing my posts, I will try once again.
@v_canoz, @i_kamal - Summary of what I found:

1. You create tasks with illegal priority configMAX_PRIORITIES, max legal is configMAX_PRIORITIES-1
   Creates big problems, fixed in my tests.
2. I reproduced your problem on LPC55S28, problem is not LPC55S69.
3. Your test works fine for me on K64F with GCC Oct-2019, FreeRTOS 10.0.1, newlib 3.1.0

Please tell us version numbers that work for you (FreeRTOS, GCC, newlib)!
We need to zero in on which is the breaking change.

Thanks,
Best Regards, Dave

[i_kamal]

Adding to Victor's post, with whom I've been working on that issue.

I confirm that the EXACT same project work flawlessly on McuXpresso V11.1.1, and crashes immediatly when compiled on McuXpresso V12.0.

The only significant changes - looking at the release notes - are:

* Upgraded: GNU ARM Embedded Toolchain to GCC9-2019-q4-major.
* Upgraded: GNU make 4.2.1 is now integrated on IDE on all OS-es.

@davenadler From our observations, I don't think it's a FreeRTOS port issue, as the whole FreeRTOS kernel (and all its files) is the same in both tests mentioned above (the test that fails and the test that functions properly).

What puzzles me also is that you (Dave) are saying that with GCC9-2019-q4-major (the same version we are suspecting of being the culprit) seems not be causing any issues in your K64 test.

Perhaps you can do the test yourself and see how switching to V11.1 fixes everything! it's quite spectacular actually! 

[davenadler]

@i_kamal- Good to know the FreeRTOS version is not the issue.

Can you please give us version numbers for what you have working: GCC and newlib?

[v_canoz]

Hi @davenadler 

The last version of GCC that works is (McuXpresso v11.1.1) 

• arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 8-2019-q3-update) 8.3.1 20190703 (release) [gcc-8-branch revision 273027]
• #define _NEWLIB_VERSION "3.1.0"
• FreeRTOS Kernel V10.3.0

One of the versions that does not work is (McuXpresso v11.2.1)

• arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 9-2019-q4-major) 9.2.1 20191025 (release) [ARM/arm-9-branch revision 277599]
• #define _NEWLIB_VERSION "3.1.0"
• FreeRTOS Kernel V10.3.0
  
  
  

Please note that the only difference is the GCC version. I hope NXP will investigate this issue quickly!

[v_canoz · Accepted solution]

We finally found the culprit of all the problems we've encountered : configENABLE_FPU is set to 0 in FreeRTOSConfig.h provided in the examples from NXP.

Whether we use the HARD FPU or the SOFT FPU, this flag enables FPU thread safety, you must definitely set it to 1 with Cortex M33, even if you don't do obvious float maths.

We think the reason why the newest version of McuXpresso crashed with our example is that most probably the snprintf function uses float operations even if we pass integers to it.

Now we do not have any problem with any one of the McuXpresso versions since we changed this flag.

To NXP : for the next time, please check that your minimum examples (aka hello world examples) are properly configured, this will allow your customers (us for example, Ikalogic) to spend time on making great things, thus buying your MCUs, instead of looking for very complicated bugs for weeks.

@davenadler thank your for your help throughout this bug resolving !

[davenadler]

Hi @v_canoz - I emailed you but here's a summary of what I have found so far:

1. You create tasks with illegal priority configMAX_PRIORITIES.
   The maximum legal priority is configMAX_PRIORITIES - 1
   This will absolutely cause big problems (I fixed this for my tests).
2. I have reproduced your problem on the LPC55S28, so it is not related to the dual-core on your '69.
3. Disabling FreeRTOS security code does not help (configRUN_FREERTOS_SECURE_ONLY 0).
4. Your problem does not appear on a Kinetis K64F using:
   1. FreeRTOS Kernel V10.0.1,
   2. arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 9-2019-q4-major) 9.2.1 20191025 (release) [ARM/arm-9-branch revision 277599]
   3. _NEWLIB_VERSION "3.1.0"

I suspect (but cannot yet prove) that this is a problem with the FreeRTOS M33 port.
Most likely it is neither a newlib nor free storage issue.

Can you verify your test cases work with an earlier version?
If so please tell us version numbers for GCC, FreeRTOS, and newlib (as this is not an MCUXpresso IDE issue)?

More later,
Best Regards, Dave