LLM Edge Studio

[#1]

LLM Edge Studio

In this post, I want to share a quick walkthrough of LLM Edge Studio, an NXP launcher application designed to test supported Large Language Models running locally on i.MX platforms with Ara240 DNPU acceleration.

LLM Edge Studio provides a simple GUI to select a model, load it, enter prompts, and interact with an LLM directly at the edge. It communicates with the Ara240 Runtime SDK through the eIQ AAF Connector, using a REST-based interface for prompt submission and streaming token responses.

 

Key Features

• Local LLM inference on supported i.MX platforms
• Ara240 DNPU acceleration
• GUI-based model selection and prompt input
• Streaming token output
• Integration with eIQ AAF Connector and Ara240 Runtime SDK

 

Supported Models

• Qwen2.5-coder-1.5B
• Qwen2.5-7B-Instruct

These models are provided as Ara240-compatible model.dvm files and are intended for local execution on the target platform.

 

Running LLM Edge Studio

Start the application with:

run_llm_edge_studio

Before launching, make sure the Ara240 runtime service is running:

systemctl status rt-sdk-ara2.service --no-pager -l

Once the GUI appears, click LOAD to load the selected model. After the model is ready, enter a prompt and submit it to start interacting with the LLM.

 

Walkthrough Video

In the attached video, I show how to launch LLM Edge Studio from GoPoint, load a supported model, submit a prompt, and view the generated response running locally on the i.MX platform with Ara240 DNPU acceleration.

 

Summary

LLM Edge Studio is a useful tool for quickly evaluating local LLM inference on NXP i.MX platforms using Ara240 DNPU acceleration. It provides a simple workflow for model loading, prompt testing, and observing token streaming directly at the edge.

Link

• LLM Edge Studio repository:
  https://github.com/nxp-imx-support/llm-edge-studio