MQX 4.2 SPI halts on K65

[yoramshacham]

I running an MQX 4.2 system over K65 with the following resources:

• Internal ADC sampling at 8K per sec.
• External ADC connected as a single SPI slave, sampling at 2K per sec (baud rate 750K).
• TCP server sending ~45 pps (each packet ~1K long) to a single client, over 100M ETH
• Shell over UART

The external SPI ADC raises an interrupt upon data ready.

I'm seeing as issue where SPI losses data and eventually would hangs in the middle of a read transaction.
Looking at the SPI lines show that CLK just stops working, causing everything to halt.

If the system does not send TCP data, the SPI works perfectly.

Only workaround I found was to change the SPI interrupt priority above all other peripherals and the SPI handling task above all other tasks.

It seems like MQX is loosing SPI interrupts for some reason.

Any idea how to fix that?

[yoramshacham]

leaving it to run for several hours, it did hung eventually.

The only thing that allows it to run freely is having the handling task at priority 7 (although it is not recommended).

any idea?

[Carlos_Musich · Accepted solution]

Hi,

doing some investigation I found that this has to do with the fact that the default DSPI_FIFO_DEPTH used is 4. Only SPI0 in the K65 has a FIFO depth of 4 the rest have 2 and 1 respectively.

Probably setting the DSPI_FIFO_DEPTH macro to 1 or 2 in the user_config.h may resolve the problem. E.g. The SPI example in MQX path uses SPI2. so setting the macro to 1 should solve the problem.

Regards,
Carlos

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[yoramshacham]

Hi Carlos,

It seems that the FIFO depth is the reason for the SPI hang.

I've tested it for about 48 hours now with sample rate of 4K/sec and it seems to work very well.

Thanks for all you assistance.

[Carlos_Musich]

Sorry for the delay,

can you please send a screen capture of the TAD > Task summary / Task list taken when the task is stuck?

Regards,

Carlos

[Carlos_Musich]

HI yoramshacham,

if a multi-tasking application is poorly designed it could fall in any of the following undesirable situations:

• Race Condition
• Priority Inversion
• CPU Time Starvation

Your application may be falling in any of these conditions and it sounds normal that raising priority solves the problem. However you must be sure that raising priority is not causing same problem in other tasks and also be aware that in MQX applications user tasks MUST NOT use priority 7 or higher (in this case a higher priority is a lower number).

I could add some additional comments if you share information about all your tasks and interrupt priorities as well as a capture of the TAD Task Summary window when application fails.

Regards,

Carlos

[yoramshacham]

Hi Carlos,

Thanks for your answer.

The system is based on HVAC2 (USB and MFS removed as are not required).
I don't think there is a race condition or deadlocks in this scenario, as tasks do not share resources and each is individual.

There was no issue of CPU starvation as the entire system continued to work as expected and all other tasks functioned correctly.

Looking at the specific task showed it is waiting on SPI event to complete (internally in SPI MQX code).
There was no apparent reason to suspect there was lack of data as the external SPI-ADC raised Data Ready interrupt.

[Carlos_Musich]

Is there a way I can reproduce this behavior on my side i order to investigate?

[yoramshacham]

Reproducing the same exact configuration is not doable, as we're using a very specific external ADC, plus I can not share the code.

I guess it can be reproduced by adding a task to HVAC2 that constantly reads SPI interface.
I find that in my configuration, it would halt almost immediately if BSP_DSPI_INT_LEVEL is greater than 3.

Please see attached the deadlock point: 

Could there be some kind of race between the DSPI interrupt and the EVENT_IO_FINISHED ?