topic Re: Glow vs TFLM performance benchmarks on i.MX RT1060? in eIQ Machine Learning Software

Glow vs TFLM performance benchmarks on i.MX RT1060?

ramkumarkoppu_p — Tue, 22 Mar 2022 14:34:45 GMT

I was reading NXP's blog on Glow compiler for PyTorch at https://medium.com/pytorch/glow-compiler-optimizes-neural-networks-for-low-power-nxp-mcus-e095abe14942

and this blog says i.MX RT1060 used to run Glow and Tensorflow lite for Micro (TFLM) frameworks. Are there any recorded performance benchmarks (inference time, memory consumption, supported operations, etc) to compare Glow and TFLM on the same device? Can you please share it if you have it?

Re: Glow vs TFLM performance benchmarks on i.MX RT1060?

david_piskula — Fri, 25 Mar 2022 08:21:15 GMT

Hello @ramkumarkoppu_p,

one such comparison can be found in AN13562

However, the results are very model dependent. In general, it has been my experience that GLOW outperformed TF Lite Micro slightly but there have also been cases where the situation was reversed. It all depends on the specific layers used in the model, what versions are supported etc. Ultimately, it just comes down to experimentation.

Since it's possible to convert TensorFlow models to ONNX and then compile them with GLOW, you can always train your model in TensorFlow and then compare the performance by converting it both to TF Lite and with GLOW and decide for yourself, which suits your project better.

As for supported operations, that depends on the version of TensorFlow/GLOW currently supported in the SDK. Please have a look at the user guides in the doc folder in the SDK.

I also found this article: https://towardsdatascience.com/tflite-micro-vs-glow-aot-6524be02ba2a which explains the difference between GLOW and TF Lite Micro pretty nicely. It's almost a year old so the data might be outdated at this point but the general idea presented in the article still applies.

Best Regards,

David

Re: Glow vs TFLM performance benchmarks on i.MX RT1060?

ramkumarkoppu_p — Fri, 25 Mar 2022 12:00:34 GMT

Thanks David for the response. My main issue is, I would like to use Research models (not yet standard) created by PyTorch but could not find PyTorch version of inference engine similar to what we have TF Lite Micro for Tensorflow to run on Cortex-M. The closest I find is Glow which seems to have PyTorch support. But I don't know if Glow can take advantage of NPUs like ARM Ethos which NXP is also planning to have it on new versions of MCUs.

Re: Glow vs TFLM performance benchmarks on i.MX RT1060?

david_piskula — Fri, 25 Mar 2022 14:13:58 GMT

I see, in that case I would definitely recommend you give GLOW a try. As for the accelerator support, you could have a look at the RT685, which offers neural network acceleration using the HiFi4 DSP and is fully supported by GLOW.

We do plan on supporting NPUs in future devices with our eIQ enablement and will be sharing more details on that in the future.

Re: Glow vs TFLM performance benchmarks on i.MX RT1060?

ramkumarkoppu_p — Mon, 28 Mar 2022 19:19:09 GMT

Hi David,

I have a TF Lite model which is targeted to run on Google Coral Dev Board. If I run the same TF Lite model on ARM Ethos-U65 microNPU which will be available on i.MX 93 application processor, will there be a performance degradation? Has NXP done any tests with ARM Ethos-U65 microNPU in terms of Inference time, operators availability, power usage?

Re: Glow vs TFLM performance benchmarks on i.MX RT1060?

david_piskula — Tue, 29 Mar 2022 05:50:24 GMT

Hello @ramkumarkoppu_p,

I am not able to comment on future releases, sorry. The i.MX 93 is still in preproduction and all its specifications are still subject to change without notice, as stated on its NXP page.

Also, since this is a thread about GLOW vs TFLM on i.MX RT1060, could you please open a new thread in case you have questions that don't relate to the topic here? It helps keep the community easier to navigate. (And since I'm focused on RT devices, I am not able to provide proper i.MX application processors support anyway.)

Best Regards,

David