2302332_en-US

はい

Long Neutron setup time for large TFLite model on Neutron NPU on i.MX95

Hi NXP Team,

We have a customer who is currently working on deploying a TensorFlow Lite model (using ResNet-50) on i.MX95 SoC using Neutron/NPU acceleration and would appreciate some guidance regarding long setup times. Here are some details:

They are using BSP based on NXPLinux 6.6.101
Input resolution320 × 320 px.
With this configuration, the model conversion works correctly and 76 out of 87 nodes are delegated to the NPU.
Inference time: ~160 ms (which is very good for them).
Accuracy of the output looks correct and meets their requirements.

However, since this model is significantly large, the setup time is extremely long (~400 seconds) before inference can start (see attached image). This seems to happen during startup, before the first inference. In a production scenario, it would force users to wait 6–7 minutes after boot before ML functionality becomes available, which is not acceptable long term.

They also observed that using lower input resolutions (e.g. 224 × 224 px) significantly reduces the setup time. However, they have not yet managed to train a model that converges reliably at this lower resolution.

Questions:

We suspect the long setup time is caused by the Neutron converter converting the model at runtime.
- Since the model does not change between boots, is there a way to cache the converted model and reuse it to reduce startup time? I find in the machine learning user guide from NXP that the iMX8 series and also for iMX93 ethos series support caching for GPU/NPU (explained in section 7.1.3 and 7.2.6.4) to improve the hardware accelerators warmup time. Is there something similar for Neutron?
Are there any recommended optimizations or best practices to reduce Neutron/NPU setup time for large models?

Any suggestions or insights would be greatly appreciated.

Re: Long Neutron setup time for large TFLite model on Neutron NPU on i.MX95

Hi @rudhinair,

Thank you for contacting NXP Support!

The issue may be related to the BSP version and the silicon revision your customer is using.
To achieve better performance, we recommend using the latest BSP versions (6.18 or 6.12), but please note that these are only supported on B0 silicon.

I strongly recommend using the most recent versions, as they provide broader compatibility and improved support for additional NPU operations.

Best regards,
Chavira

Re: Long Neutron setup time for large TFLite model on Neutron NPU on i.MX95

Hi @Chavira,

Thanks for your answer. You are right, in the meantime, the customer tested on the newer BSP with kernel 6.12, and I can confirm that the issue has been solved. Thanks for your support!

Re: Long Neutron setup time for large TFLite model on Neutron NPU on i.MX95

Hi, was this issue resolved? Could someone from NXP please explain the difference in Silicon Revisions and how they impact NPU operations/support and BSP support.