2402547_en-US

[#1]

NPU support on the i.MX95 Verdin EVK

Hi,

I am trying to run my tflite models on the i.MX95 NPU. The models can be converted, I see the acceleration using the benchmark but the output is not usable at all (always the same for several face detection and face landmark models)

I then tried to run the example according to this user guide: https://www.nxp.com/docs/en/user-guide/UG10166.pdf

root@imx95-19x19-verdin-47:/usr/bin/tensorflow-lite-2.19.0/examples# ./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libneutron_delegate.so
INFO: Loaded model mobilenet_v1_1.0_224_quant.tflite
INFO: resolved reporter
INFO: EXTERNAL delegate created.
INFO: NeutronDelegate delegate: 1 nodes delegated out of 4 nodes with 1 partitions.

INFO: Neutron delegate version: v1.0.0-f24d08e5, zerocp enabled.
INFO: Applied EXTERNAL delegate.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: invoked
INFO: average time: 0.37 ms

-> as you can see the inference ran fine but there is no classification as mentioned in the user guide to check the actual working of the model.

I converted the model using the correct converter version of SDK 2.2.2:

NeutronSDK_2.2.2+LF_6.12.49_2.2.0/neutron-converter --input input/mobilenet_v1_1.0_224_quant.tflite --target imx95 --output output/mobilenet_v1_1.0_224_quant.tflite --dump-statistics          

Performance estimates:
        Clock Frequency: 0.000000 MHz
        Clock cycles per inference: 0
        Latency per inference: -nan ms 
        Inferences per second: -nan

Memory  footprint: 
        Variables size: 0.000000 MB
        Constants size: 0.000000 MB
        Microcode size: 0.000000 MB

Statistics for NeutronGraph "subgraph_030":
  Operators:
    Number of Neutron operators = 29
    Number of builtin operators = 44
  Memory:
    Inputs        = 150,528 (bytes)
    Microcode     = 23,944 (bytes)
    Weights       = 4,329,648 (bytes)
    Kernels       = 11,088 (bytes)
    Outputs       = 381,913 (bytes)
    Scratch       = 380,912 (bytes) (Allocation efficiency: 1)
    Total data    = 913,353 (bytes) (Inputs + Outputs + Scratch)
    Total weights = 4,364,680 (bytes) (Microcode + Weights + Kernels)
    Total size    = 5,278,033 (bytes) (All)
  Latency:
    Cycle estimation   = 1,066,681 (cycles)
    Latency estimation = 1.067 (ms) (@ 1000.000 MHz)

Overall statistics for graph "":
  Operators:
    Number of operators after import    = 31
    Number of operators after optimize  = 47
    Number of operators after extract   = 4
      Number of Neutron graphs          = 1
      Number of operators total         = 47
      Number of operators converted     = 44
      Number of operators NOT converted = 3
    Operator conversion ratio           = 44 / 47 = 0.93617
    Operators converted                 = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,
  Memory:
    Total data    = 532,448 (bytes) (Inputs + Outputs + Intermediate Variable Tensors)
    Total weights = 4,364,688 (bytes) (Weights)
    Total size    = 4,897,136 (bytes) (All)
  Latency:
    Cycle estimation   = 1,066,681 (cycles) (NPU only)
    Latency estimation = 1.067 (ms) (@ 1000.000 MHz) (NPU only)

Conversion time:
  Optimization = 2.00387 (seconds)
  Extraction   = 0.0308523 (seconds)
  Generation   = 5.56016 (seconds)
  Total        = 7.59489 (seconds)

My hardware and BSP setup:

SOC: iMX Verdin EVK SoM V1.0C

Carrier: iMX Verdin EVK v1.2A (https://www.toradex.com/de/computer-on-modules/verdin-arm-family/nxp-imx95-evaluation-kit?srsltid=Af...)

For the build I am using:

MACHINE = "imx95-19x19-verdin"

I am using this BSP version: https://www.toradex.com/de/news/bsp-layers-reference-images-walnascar?srsltid=AfmBOoqJF2YHULiIe7sA1L...

And the NXP layers:

meta-imx

rel_imx_6.12.49_2.2.0

with kernel version:

6.12.49-lts-next-g759f4038100f

-> My main question is if my silicon revision (A0) is even supported or what could be the issue when running the models fails silently and just produces garbage output:

Face detection (UltraFace-Ultraslim, [1,128,128,3] int8/uint8 input) outputs a single fused (172,6) tensor of [bg_score, face_score, xmin, ymin, xmax, ymax] per anchor, already NMS'd and normalized to [0,1] — on CPU this yields one clear high-confidence detection (~0.996) tightly bounding the face, while on NPU all 172 anchors collapse to an identical constant (~0.50 score, near-zero-size box at ~0.227,0.227,0.227,0.227). Face landmarks (NXP facial_landmarks_35, [1,60,60,3] uint8 input) output a [1,70] tensor of 35 interleaved (x,y) points normalized to the face crop — on CPU these form a recognizable face-point pattern when overlaid on the image, while on NPU the entire 70-value output likewise collapses to a single repeated constant instead of varying per point.

Yocto ProjectRe: NPU support on the i.MX95 Verdin EVK

output with verbose:

root@imx95-19x19-verdin-4798be6ce85542d2:/usr/bin/tensorflow-lite-2.19.0/examples# ./label_image -m demo_converted -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libneutron_delegate.so -v 1 -r 5
INFO: Loaded model demo_converted
INFO: resolved reporter
INFO: tensors size: 11
INFO: nodes size: 4
INFO: inputs: 1
INFO: input(0) name: input
INFO: 0: MobilenetV1/Logits/SpatialSqueeze, 1001, 9, 0.166099, -62
INFO: 1: MobilenetV1/Predictions/Reshape_1, 1001, 3, 0.00390625, 0
INFO: 2: input, 150528, 3, 0.0078125, 128
INFO: 3: MobilenetV1/Predictions/Reshape_1/requantize, 1001, 9, 0.00390625, -128
INFO: 4: input, 150528, 9, 0.0078125, 0
INFO: 5: NeutronMicrocode, 23944, 3, 0, 0
INFO: 6: NeutronWeights, 4329648, 3, 0, 0
INFO: 7: NeutronKernels, 11088, 3, 0, 0
INFO: 8: NeutronScratch, 380912, 3, 0, 0
INFO: 9: NeutronProfile, 0, 3, 0, 0
INFO: 10: NeutronDebug, 0, 3, 0, 0
INFO: len: 940650
INFO: width, height, channels: 517, 606, 3
INFO: input: 2
INFO: number of inputs: 1
INFO: number of outputs: 1
INFO: EXTERNAL delegate created.
INFO: NeutronDelegate delegate: 1 nodes delegated out of 4 nodes with 1 partitions.

INFO: Neutron delegate version: v1.0.0-f24d08e5, zerocp enabled.
INFO: Applied EXTERNAL delegate.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Interpreter has 1 subgraphs.

-----------Subgraph-0 has 11 tensors and 5 nodes------------
1 Inputs: [2] -> 150528B (0.14MB)
1 Outputs: [1] -> 1001B (0.00MB)

Tensor  ID Name                      Type            AllocType          Size (Bytes/MB)    Shape      MemAddr-Offset  
Tensor   0 MobilenetV1/Logits/Spa... kTfLiteInt8     kTfLiteCustom      1001     / 0.00 [1,1001] [-1, -1)
Tensor   1 MobilenetV1/Prediction... kTfLiteUInt8    kTfLiteArenaRw     1001     / 0.00 [1,1001] [151552, 152553)
Tensor   2 input                     kTfLiteUInt8    kTfLiteArenaRw     150528   / 0.14 [1,224,224,3] [0, 150528)
Tensor   3 MobilenetV1/Prediction... kTfLiteInt8     kTfLiteArenaRw     1001     / 0.00 [1,1001] [150528, 151529)
Tensor   4 input                     kTfLiteInt8     kTfLiteCustom      150528   / 0.14 [1,224,224,3] [-1, -1)
Tensor   5 NeutronMicrocode          kTfLiteUInt8    kTfLiteMmapRo      23944    / 0.02 [23944] [4340768, 4364712)
Tensor   6 NeutronWeights            kTfLiteUInt8    kTfLiteMmapRo      4329648  / 4.13 [4329648] [11104, 4340752)
Tensor   7 NeutronKernels            kTfLiteUInt8    kTfLiteMmapRo      11088    / 0.01 [11088] [0, 11088)
Tensor   8 NeutronScratch            kTfLiteUInt8    kTfLiteArenaRw     380912   / 0.36 [380912] [-1, -1)
Tensor   9 NeutronProfile            kTfLiteUInt8    kTfLiteArenaRw     0        / 0.00 [0] [-1, -1)
Tensor  10 NeutronDebug              kTfLiteUInt8    kTfLiteArenaRw     0        / 0.00 [0] [-1, -1)

kTfLiteArenaRw Info: 
Tensor 2 has the max size 150528 bytes (0.144 MB).
This memory arena is estimated as[0xaaaafd73ffa9, 0xaaaafd71abc0), taking 152553 bytes (0.145 MB).
One possible set of tensors that have non-overlapping memory spaces with each other, and they take up the whole arena:
Tensor 2 -> 3 -> 1.

kTfLiteArenaRwPersistent Info: not holding any allocation.

kTfLiteMmapRo Info: 
Tensor 6 has the max size 4329648 bytes (4.129 MB).
This memory arena is estimated as[0xffff7ec299f8, 0xffff7e800050), taking 4364712 bytes (4.163 MB).
One possible set of tensors that have non-overlapping memory spaces with each other, and they take up the whole arena:
Tensor 7 -> 6 -> 5.

kTfLiteDynamic Info: not holding any allocation.

=== Beginning of kTfLiteArenaRw Dump: ===
Total size is 152553 bytes (0.145 MB), holding 3 tensors.
tensor 2: life_span: node [0, 4], size:  150528 bytes (0.144 MB).
tensor 3: life_span: node [2, 3], size:  1001 bytes (0.001 MB).
tensor 1: life_span: node [3, 4], size:  1001 bytes (0.001 MB).
1 tensors are of same max size (150528 B (0.144 MB)): [2]
Per-layer-info in the order of op execution:
Node 0: 150528 bytes (0.144 MB), utilization rate: 98.673%, 1 live tensors: [2]
Node 4: 151529 bytes (0.145 MB), utilization rate: 99.329%, 2 live tensors: [1,2]
Node 2: 151529 bytes (0.145 MB), utilization rate: 99.329%, 2 live tensors: [2,3]
Node 3: 152530 bytes (0.145 MB), utilization rate: 99.985%, 3 live tensors: [1-3]

Top 4 memory-consuming layers:
Node 3: 152530 bytes (0.145 MB), utilization rate: 99.985%, 3 live tensors: [1-3]
Node 4: 151529 bytes (0.145 MB), utilization rate: 99.329%, 2 live tensors: [1,2]
Node 2: 151529 bytes (0.145 MB), utilization rate: 99.329%, 2 live tensors: [2,3]
Node 0: 150528 bytes (0.144 MB), utilization rate: 98.673%, 1 live tensors: [2]

===End of kTfLiteArenaRw Dump: ===

Node   0 Operator Builtin Code 114 QUANTIZE (not delegated)
  1 Input Tensors:[2] -> 150528B (0.14MB)
  1 Output Tensors:[4] -> 150528B (0.14MB)
Node   1 Operator Custom Name NeutronGraph (delegated by node 4)
  4 Input Tensors:[4,5,6,7] -> 0B (0.00MB)
  4 Output Tensors:[0,8-10] -> 0B (0.00MB)
Node   2 Operator Builtin Code  25 SOFTMAX (not delegated)
  1 Input Tensors:[0] -> 1001B (0.00MB)
  1 Output Tensors:[3] -> 1001B (0.00MB)
Node   3 Operator Builtin Code 114 QUANTIZE (not delegated)
  1 Input Tensors:[3] -> 1001B (0.00MB)
  1 Output Tensors:[1] -> 1001B (0.00MB)
Node   4 Operator Custom Name NeutronDelegate 
  4 Input Tensors:[4-7] -> 4515208B (4.31MB)
  1 Output Tensors:[0] -> 1001B (0.00MB)

Execution plan as the list of 4 nodes invoked in-order: [0,4,2,3]
Among these nodes in the execution plan:
  Node 4 is a NeutronDelegate node (0xaaaafd6f5c30), which has delegated 1 nodes: [1]
--------------Subgraph-0 dump has completed--------------

--------------Memory Arena Status Start--------------
Total memory usage: 152553 bytes (0.145 MB)
- Total arena memory usage: 152553 bytes (0.145 MB)
- Total dynamic memory usage: 0 bytes (0.000 MB)

Subgraph#0   Arena (Normal)         152553 (100.00%)
--------------Memory Arena Status End--------------

INFO: invoked
INFO: average time: 0.326 ms