UART Rx (callback) Makes MQX Weird

[jeffreygraves]

If I send enough characters to the UART, I can make MQX act weird*.

*MQX will stop running a task or...

*MQX will stop running more than one task or...

*MQX will run a task open loop, ignoring wait functions (ie OSA_TimeDelay()).

I have attached a "bare bones" project that reproduces this behavior.

This project uses a UART Rx data callback function.

(This simple project does not do anything with the received chars.)

This project targets the FRDM-K22F development board.

This project uses the following software:

KDS 2.0

KSDK 1.1.0

MQX-KSDK

PEx

The project configuration is straight forward and what you get with PEx.

I have not taken any steps to install other interrupt handlers.

Steps to reproduce:

Comm settings: 115Kbps,N,8,1

Monitor the referenced I/O pins...

Repeatedly paste a string (~80 char) into a terminal app like Tera Term.

Eventually, the toggling I/O (square wave) will change indicating a MQX problem.

I have reproduced this problem with and without a Tera Term character delay (1ms).

Has anyone else seen this problem?

This issue is affecting the stability of a more complicated product.

Any feedback would be greatly appreciated.

Thanks,

Original Attachment has been moved to: UART_Rx_Makes_MQX_Wierd01.zip

[DavidS]

Hi Jeffrey,

Looking at your code and don't have hardware to run on right now but can test later.

Please try changing to uartCom2RxBuff[2] in os_tasks.c.

Also in your Cpu:MK22FN512VLH12 Component Inspector - Cpu window, please set the Init clock configuration to "1 - Maaximum achievable clock frequency configuration".  You have it set to Default with has the CPU frequency of 20.9MHz and the UART operating at 115200 baud which really taxes the system with an RTOS doing other stuff.

Regards,

David

[jeffreygraves]

David,

I tried the changes that you suggest.  It does seem harder to force the failure with the faster CPU frequency.  However, I can still make it happen.

Also, if I try to do a UART_DRV_SendData or UART_DRV_SendDataBlocking in Task1_task, MQX really "gets weird".

I even tried bumping the OSA_TimeDelay in Task1_task up to 3000ms...

Does this offer any kind of clue?

I guess I will try to configure the "debug_console" and use "_int_install_unexpected_isr".

Does this sound like a reasonable approach to debug?

Thank you for looking into this.

Best regards,

[jeffreygraves]

David,

Did you get a chance to try this on a FRDM-K22F?

I am having some trouble with the MQX debug_console.

I could really use some help...

[DavidS]

Hi Jeffrey,

Sorry for delay.  Work gets in the way of my work.

Attached is my ZIPed project.  It can be un-ZIPed into the KSD_1.1.0 folder.

I added a gpio toggle to the UART Rx Callback too.

I'm able to download 129 bytes consistently without failure from within Hercules terminal.

I also have downloaded 1025 byte file too without failure and included it in the ZIP file.

Odd byte count so the Blue LED will alternately end up on/off.

Please review and test and let me know what happened on your hardware.

Hopefully this helps.

Regards,

David

[jeffreygraves]

David,

I am stuck.  This is a late dev cycle show stopper issue for us.  I really need a fix or a work around...

Andy suggested that I add the ability to "suspend/resume" Task1 and Task2 via the debugger.

In the failure condition with one task stopped, he thought it would be helpful to know if that task could be re-started...

I will try to add this this now and post the findings.

Thank you,

[Martin_ · Accepted solution]

Hi Jeffrey,

in one of the generated files Cpu.c there is something very strange. See the code snippet below on the line (09).

#if CPU_COMPONENTS_INIT
void Components_Init(void)
{

    /*! uartCom1 Auto initialization start */
    OSA_InstallIntHandler(UART1_RX_TX_IRQn, UART1_MSJ_IRQHandler);
    UART_DRV_Init(FSL_UARTCOM1,&uartCom1_State,&uartCom1_InitConfig0);
    UART_DRV_InstallRxCallback(FSL_UARTCOM1, uartCom1_RxCallback, uartCom1RxBuff, NULL, true);
    NVIC_SetPriority(UART1_RX_TX_IRQn, 80U);
    /*! uartCom1 with DMA Auto initialization end */
  /*! gpio1 Auto initialization start */
  GPIO_DRV_Init(NULL,gpio1_OutConfig0);
  /*! gpio1 Auto initialization end */
}
#endif /* CPU_COMPONENTS_INIT */

The second argument passed to NVIC_SetPriority is 80U which is very strange. When we look into core_cm4.h implementation, we will see the value is left shifted by some number of bits (I guess by 4 on Kinetis as it has 4 interrupt priority bits) and then there is binary AND with 0xff and finally the result is written into an array of uint8_t in NVIC->IP[]. So I'm afraid this operation actually produces zero that is written to NVIC priority. This is actually above the MQX critical section priority (when MQX enters critical section it masks interrupts priority level 2 and lower, so an isr with zero priority can interrupt MQX critical section).

Now if you're using blocking UART_DRV methods to receive data,  the UART_DRV interrupt routine would call semaphore api inside (post rx semaphore when all requested chars have been received).

This can corrupt MQX scheduler context I'm afraid, as the semaphore post migth be called inside MQX critical section.

By making the long story short, I'd recommend to try with something like NVIC_SetPriority(UART1_RX_TX_IRQn, 6); (use even priority level lower than 2).

Martin

[jeffreygraves]

Happy Dance!!!!  Thank you!!!:smileyhappy::smileyhappy:

This seems to be a fix for our real product!

So far it seems very stable!

Our real product uses both fsl_uart and fsl_i2c PEx components.

It seems like BOTH may wind up setting invalid NVIC priorities.

I used 8U and 6U to keep above 2 x MQX_HARDWARE_INTERRUPT_LEVEL_MAX as called for in MQXKSDK110RN - Known Issues.

8U and 6U also occupy the lower nibble and survive the shift and AND ing you describe above.

Do these values seem reasonable to you?

(Components_Init() screenshots are attached)

Here are the values I hand edited for testing/verification:

NVIC_SetPriority(UART0_RX_TX_IRQn, 8U);

NVIC_SetPriority(UART1_RX_TX_IRQn, 8U);

NVIC_SetPriority(I2C0_IRQn, 6U);

NVIC_SetPriority(I2C2_IRQn, 6U);

We have dealt with a lot of strange behavior up to this (late) point.

I am hopeful now that we can focus on what our product needs to do in a more straight forward manner.

[DavidS]

My guess is the 80 should be hex 0x80.

So upper nibble used and lower ignored to fit into the NVIC IPR IRQx field.

Do I think the PE field for setting the UART interrupt priority level needs fixing.

Regards,

David

Sent from my iPhone

[DavidS]

Awesome find Martin.

Regards,

David

Sent from my iPhone

[jeffreygraves]

Martin,

Very promising!  This seems to have fixed the problem in our FRDM-K22 test app.

I will now switch back to our real product and test again.

This issue seems to be mentioned in the MQX-KSDK 1.1.0 Release Notes (MQXKSDK110RN).

I should mention that considering the bit shifting you describe above, it is impossible to select a good NVIC priority value using the fsl_uart PEx component...

Should we report this as a bug?

I can live with this for now...

I will let you know soon how this works in our real product.

Thank you!!!!!!

[dave408]

Wow, I'm so glad you were able to get a resolution for this problem.  Something tells me that I would have run into the exact same problem in the not-too-distant future and it would have been a real showstopper for me as well!

[jeffreygraves]

David,

I can still make one of the tasks stop running.  To do so, I have to copy a line of text (~80 char) and continually paste it into Tera Term (hold down alt-V).  In 10 sec or so, one of the tasks will stop toggling I/O and not recover.

Our real application already had the CPU clock set pretty high (96MHz) I bumped it up to 120Mhz, but I still see the problem.  It is much easier to re-produce this problem with our real project/hardware perhaps because the CPU is busier.

I'm afraid I am still stuck.

Is there a way for the MCU to recover from this kind of fault?

Are there any other ways to avoid this fault (ie priorities)?

Is this a known issue?

Will KSDK 1.2 help?

Should I try to switch to MQX Lite?

There was some?? reason I didn't do this to begin with...

Thank you,

[jeffreygraves]

I am still stuck.

I had a visit from Andy Juengst, a former Freescale FAE and current Mfg Rep (Freescale) with m2stech.

He suggested that I post some screen shots of MQX task aware debugger tabs while the tasks are running and then after one of the tasks has stopped.

They are attached.

Andy thought it important that I point out the following:

• In the failure condition, one or both of the tasks have stopped.
• I can tell when a task stops because I am monitoring I/O with an O-Scope.
• The UART Callback function is still being called because I can still see the RGB LED toggle color when a char is received.

Any other suggestions?

Can I provide more information?

FYI:

I tried switching this test app to MQX Lite - same problem

I tried changing the UART interrupt priority - same problem

Thank you,