Interrupting the Glow bundle inference function

[flee-elemind]

Hello!

We are using the Glow compiler to generate some CNN network bundles to perform inference on the imxrt685 cpu. This compilation process works fine with MCUXpresso. We are running the ARM CM33 port of FreeRTOS

However, the inference, while it runs fine, seems to be uninterruptible on a single cm33 core; we are trying to perform a variety of other tasks (playing audio, data collection) during the inference so this creates a problem where these other tasks are completely blocked during the inference time.

We are attempting to avoid using the HiFi core due to power consumption requirements.

Is there any way for us to allow for the ML inference to yield so that other tasks can be serviced?

Florence

[flee-elemind]

Thanks for the detailed response! Do you know if the tflite micro compiler does support task preemption, as an alternative?

Otherwise I agree that the Hifi might be the way to go…

[jingpan]

Hi @flee-elemind ,

You can use FreeRTOS or other embedded OS to preempt CPU from inference task by SysTick. Give inference task a middle level priority and other task high level priority.

Regards,

Jing

[flee-elemind]

Is there any special process required other than changing the priorities (which we have done, assigning the inference task the lowest as a test) required to allow preemption?

We have not yet been successful preempting the inference bundle. 

[jingpan]

Hi @flee-elemind ,

No, a normal task can't prevent OS from timing base schedule. Systick can interrupt current task and then OS will schedule tasks. At least freertos can make it.

Regards,

Jing

[flee-elemind]

Systick to date has not interrupted the inference function without any modifications, and I was under the impression that modifying the Systick code in the FreeRTOS port was inadvisable.

Do you know if the glow compiled code is at any point disabling the systick interrupt?

[jingpan]

Hi @flee-elemind ,

It sounds impossible. You can break the inference by debug probe, check the systick to see if it still work. And also check if the systick and global interrupt is disabled.To enable time slice switch，configUSE_PREEMPTION and configUSE_TIME_SLICING must be 1，configTICK_RATE_HZ determine the time length.

When any interrupt comes, you can use  portYIELD_FROM_ISR(pdTRUE) to switch task.

Regards,

Jing

[prakashram72]

Hi @flee-elemind ,

It sounds like you're running into a common issue when dealing with real-time systems and machine learning tasks. The Glow compiler, while efficient, does not inherently support task preemption, which is why your other tasks are being blocked during inference.

One possible solution is to manually insert task yield points into the Glow generated code. This would allow the FreeRTOS scheduler to switch tasks during the inference process. However, this would require modifying the generated code, which might not be ideal.

Another approach could be to split the inference task into smaller tasks that can be scheduled independently. This would allow other tasks to be interleaved with the inference task, reducing the blocking effect. However, this might increase the total inference time due to the overhead of task switching.

Lastly, you could consider using a separate core for the inference task, if your hardware supports it. This would allow the inference to run in parallel with your other tasks. I understand you're trying to avoid using the HiFi core due to power consumption, but it might be worth considering if the blocking issue is severe.

Here are some links to relevant discussions on the NXP community forum that might be helpful:

• How to use FreeRTOS on i.MX RT
• How to use the Cortex-M33 core on i.MX RT685
• How to use the Glow compiler with i.MX RT

I hope this helps!