LLM on i.MX93

[Behnam]

I recently received my FRDM iMX93 board and am interested in running a minimal LLM or Transformer on it. I found a document by NXP (GEN-AI-RAG-WHITEPAPER) that states TinyLlama has successfully run on iMX93, but there is no more technical information or resources available.

I know I need to ensure the model is compatible with TFLite, then quantize and compile it using eIQ and the Vela compiler, and finally run it on the iMX93 board. However, this seems like a long and complex process. I would prefer to have some kind of tutorial or at least a reliable reference to guide me through the steps.

[Behnam]

I was able to run the MobileBERT quantized model on i. MX93, but since the MatMul is not yet supported by the Vela Compiler (and u65), the performance does not seem very good. 

[Zhiming_Liu]

Hi,

The related document and toolkit are not ready to release.

Best Regards,
Zhiming

[Behnam]

Hi Zhimming

I see, so then can you share some high-level information that I can use to try to do it myself? For example, it would be helpful to know if I can use the eIQ for that or not, because I’ve tested a couple of models and it wasn’t successful for different reasons. I don’t need a full LLM right now, but a smaller language model like MobileNet would be helpful as well.

I’m looking forward to learning about the best practice for testing a small model and any details that could help me.

Best,
Behnam

[Zhiming_Liu]

Hi,

You can refer the eIQ user guide to export tflite model.

For model deploy, please refer this guide:

https://www.nxp.com/docs/en/user-guide/UG10166.pdf

Best Regards,
Zhiming