RT-DETR cannot fully convert to Ara240 .dvm on AraSDK 2.1.1 — unsupported ops + hybrid workaround Hello NXP community, I am converting RT-DETR object detectors for Ara240 DNPU on FRDM i.MX95, targeting AraSDK / runtime 2.1.1 only (not 3.0). I want to share findings and ask whether there is a recommended / supported path for full .dvm conversion of DETR-style models with multi-scale deformable attention. Environment ItemValue Board FRDM i.MX95 + Ara240 Runtime imx-nxp-ara2 2.1.1 (must match compiler) Compiler AraSDK 2.1.1 (ara-sdk-cnn:2.1.1, dvrun) Models tried Ultralytics rtdetr-l / rtdetr-x (HGNetv2); also reviewed HF PekingU/rtdetr_r50vd (ResNet-50-VD) ONNX export Ultralytics RTDETR.export(format="onnx", imgsz=640, batch=1, dynamic=False, opset=17) → FP32 static ONNX Quantization Done inside AraSDK (analyzer + calib images) → INT8 .dvm (not Ultralytics INT8 ONNX) Goal Full end-to-end conversion: FP32 ONNX → dvconvert → dryrun estimate → quantize → estimate → emit → .dvm that loads on runtime 2.1.1 and runs entirely on the Ara240 DNPU. Observation vs NXP HuggingFace Ara240 models Published Ara240 models such as Qwen2.5-VL / Qwen2.5-Coder convert successfully even though they are much larger. Those graphs are dominated by static MatMul / Norm / Softmax stacks and appear first-party ported. RT-DETR fails not because of size (~33–43M params), but because of operator coverage: the detection decoder uses multi-scale deformable attention and dynamic query selection. Unsupported / problematic layers (same class on Ultralytics and HF RT-DETR) PatternWhereEffect on AraSDK 2.1.1 GridSample / F.grid_sample Decoder MS deformable attention (n_points=4, 3 levels, 6 layers) Not a DNPU op; ONNX export produces many GridSample nodes (18 on Ultralytics rtdetr-l) TopK + gather / mask ops Uncertainty-minimal query selection (num_queries=300) Data-dependent indexing; converter falls back or schedule/emit fails AIFI self-attention Hybrid encoder Schedules as BDOT; estimator can crash (total_wgs: 0) Opset-17 ReduceSum (axes as input) Decoder attention path Converter rename_ReduceSum IndexError → DV_ATen (CPU) Identity / depth broadcasts e.g. Expand for GatherElements mask IR serialize abort (DVEEIR Broadcast layer is not needed / depth-dim broadcast) HuggingFace PekingU/rtdetr_r50vd uses the same deformable decoder + AIFI pattern (RTDetrMultiscaleDeformableAttention + grid_sample + topk in Transformers source). Only the backbone differs (ResNet-50-VD vs HGNet). Switching to HF does not make full DNPU conversion possible. Failures I hit (stage by stage) IR serialize (full graph) DVEEIR Broadcast layer is not needed at all … BCST … 300 Source: /model.28/Expand_1 (TopK mask expand). After Expand→Concat rewrite DVEEIR broadcast case is not supported in depth dimension along with other dims yet Still on the GatherElements / broadcast path. Full reconvert ReduceSum opset-17 → DV_ATen; Mod/Less/Xor/And → DV_ATen; conversion validation EMAX > 1e-4. Note: cannot drop to opset 13 because GridSample needs opset ≥ 16. Hybrid: backbone + encoder (exclude decoder only) Dryrun estimator: DVESIT Total WGs are not calculated properly at SINIT … BDOT,400,400,32 … total_wgs: 0 (AIFI). Emitter: DVEMCU Unknown CP kernels present in model … exit 139 (dummy/generic kernels after tolerating dryrun crash → no real .dvm). Hybrid: backbone only (pure CNN) Dryruns reach DVEQIR Serializing IR completed — this path is viable for a DNPU .dvm of the CNN backbone; AIFI + CCFM + decoder stay on CPU (onnxruntime). How I convert (working approach: hybrid) Because a full RT-DETR .dvm is not achievable on 2.1.1 with current op support, I use a parity-exact hybrid: Step 1 — Export FP32 static ONNX (Ultralytics) from ultralytics import RTDETR
model = RTDETR("rtdetr-l.pt")
model.export(format="onnx", imgsz=640, batch=1, dynamic=False, simplify=True, opset=17) dvrun takes this FP32 ONNX as inet/iwt. INT8 is produced later by AraSDK quantizer + calibration set. Step 2 — Extract FP32 subgraphs (ONNX extract_model) Cut at the backbone outputs (Ultralytics RT-DETR-L): Inputs: images [1,3,640,640] Outputs (DNPU / Part A): /model.3/ec/act/Relu_output_0 → [1,512,80,80] /model.7/Add_output_0 → [1,1024,40,40] /model.10/conv/Conv_output_0 → [1,256,20,20] Rest (CPU / Part B): those 3 tensors → output0 [1,300,6] from onnx import utils as onnx_utils
BOUNDARY = [
"/model.3/ec/act/Relu_output_0",
"/model.7/Add_output_0",
"/model.10/conv/Conv_output_0",
]
onnx_utils.extract_model("rtdetr-l.onnx", "partA_backbone.onnx", ["images"], BOUNDARY)
onnx_utils.extract_model("rtdetr-l.onnx", "partB_rest.onnx", BOUNDARY, ["output0"]) Part A is pure CNN (Conv/Relu/MaxPool/Concat) — no GridSample / TopK / AIFI. I verified: full(images) == partB(partA(images)) with max_abs = 0 (bit-exact). Step 3 — Compile Part A with AraSDK 2.1.1 (dvrun) YAML: srcfw: onnx, inet/iwt = partA_backbone.onnx, dim 3,640,640, calib folder, ignore_mse: true if needed Host: enough RAM/swap for estimator (I used a 32 GiB RAM / ~100 GiB swap server; 16 Gi desktop OOMs easily) Keep compiler and board runtime both on 2.1.1 Success criteria: schedule serializes, emitter reports Ara2 Processor compatibility, non-empty .dvm / assets (not 0-byte kernel.lib) Step 4 — Deploy hybrid inference Load Part A .dvm on Ara240 (runtime 2.1.1). Run Part B (partB_rest.onnx) on ARM with onnxruntime. Feed DNPU feature maps into Part B → detections. Model Part A is converted into .dvm so i have tested Part A on DNPU + PART B on CPU but it give false output low detection with zero class output but both PART A onnx and PART B onnx work on A55 cpu and gives correct output. I am attaching images for reference. ORT A + B Image Hybrid A + B Questions for NXP / Kinara Is RT-DETR / Deformable DETR on the supported / validated model list for Ara240 2.1.1? Is there a planned kernel or conversion path for GridSample / multi-scale deformable attention? For AIFI-like attention, is there a recommended ONNX rewrite or constraint so BDOT schedules without total_wgs: 0? Is a hybrid (DNPU backbone + CPU decoder) the intended pattern for this architecture until kernels exist? Any official guidance on opset (16/17) vs converter ReduceSum attribute form for DETR graphs? If others have successfully produced a full RT-DETR .dvm on Ara240 2.1.1 (or 3.x), please share: export settings (Ultralytics vs HF / lyuwenyu) YAML / dvrun flags whether deformable attention was rewritten or offloaded Happy to share more logs (estimator BCST / BDOT / emitter CP-kernel messages) if useful. Thanks, Vijay Rana Re: RT-DETR cannot fully convert to Ara240 .dvm on AraSDK 2.1.1 — unsupported ops + hybrid workaroun Hi @vijayranaACL
1. Is RT-DETR / Deformable DETR on the supported / validated model list for Ara240 2.1.1?
>>>No, This model is not currently included in the support list for Ara240.
2. Is there a planned kernel or conversion path for GridSample / multi-scale deformable attention?
>>>There are no plans at the moment.
I need to confirm with internal team regarding the other questions before getting back to you.
Could you please share the model file you tested? I can then run some tests on my i.MX95 FRDM board.
B.R Re: RT-DETR cannot fully convert to Ara240 .dvm on AraSDK 2.1.1 — unsupported ops + hybrid workaroun These are the all level of model conversion files from .pt -> .onnx -> hybrid A + B -> DVM of A + int8 onnx B. model link: https://drive.google.com/file/d/1YhxLetVxP0T85gv0qK1GT1r_T1i25f7P/view?usp=sharing
View full article