i.MX95 Verdin EVK 上的 NPU 支持 您好， 我正在尝试在 i.MX95 NPU 上运行我的 tflite 模型。这些模型可以转换，我使用基准测试看到了加速效果，但输出结果完全无法使用（对于几种人脸检测和人脸关键点模型来说，结果始终相同）。 然后我尝试按照这份用户指南运行示例： https://www.nxp.com/docs/en/user-guide/UG10166.pdf root@imx95-19x19-verdin-47:/usr/bin/tensorflow-lite-2.19.0/examples# ./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libneutron_delegate.so
INFO: Loaded model mobilenet_v1_1.0_224_quant.tflite
INFO: resolved reporter
INFO: EXTERNAL delegate created.
INFO: NeutronDelegate delegate: 1 nodes delegated out of 4 nodes with 1 partitions.
INFO: Neutron delegate version: v1.0.0-f24d08e5, zerocp enabled.
INFO: Applied EXTERNAL delegate.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: invoked
INFO: average time: 0.37 ms 如您所见，推理运行正常，但没有像用户指南中提到的那样进行分类，以检查模型的实际运行情况。 我使用SDK 2.2.2的正确变流器版本转换了模型： NeutronSDK_2.2.2+LF_6.12.49_2.2.0/neutron-converter --input input/mobilenet_v1_1.0_224_quant.tflite --target imx95 --output output/mobilenet_v1_1.0_224_quant.tflite --dump-statistics
Performance estimates:
Clock Frequency: 0.000000 MHz
Clock cycles per inference: 0
Latency per inference: -nan ms
Inferences per second: -nan
Memory footprint:
Variables size: 0.000000 MB
Constants size: 0.000000 MB
Microcode size: 0.000000 MB
Statistics for NeutronGraph "subgraph_030":
Operators:
Number of Neutron operators = 29
Number of builtin operators = 44
Memory:
Inputs = 150,528 (bytes)
Microcode = 23,944 (bytes)
Weights = 4,329,648 (bytes)
Kernels = 11,088 (bytes)
Outputs = 381,913 (bytes)
Scratch = 380,912 (bytes) (Allocation efficiency: 1)
Total data = 913,353 (bytes) (Inputs + Outputs + Scratch)
Total weights = 4,364,680 (bytes) (Microcode + Weights + Kernels)
Total size = 5,278,033 (bytes) (All)
Latency:
Cycle estimation = 1,066,681 (cycles)
Latency estimation = 1.067 (ms) (@ 1000.000 MHz)
Overall statistics for graph "":
Operators:
Number of operators after import = 31
Number of operators after optimize = 47
Number of operators after extract = 4
Number of Neutron graphs = 1
Number of operators total = 47
Number of operators converted = 44
Number of operators NOT converted = 3
Operator conversion ratio = 44 / 47 = 0.93617
Operators converted = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,
Memory:
Total data = 532,448 (bytes) (Inputs + Outputs + Intermediate Variable Tensors)
Total weights = 4,364,688 (bytes) (Weights)
Total size = 4,897,136 (bytes) (All)
Latency:
Cycle estimation = 1,066,681 (cycles) (NPU only)
Latency estimation = 1.067 (ms) (@ 1000.000 MHz) (NPU only)
Conversion time:
Optimization = 2.00387 (seconds)
Extraction = 0.0308523 (seconds)
Generation = 5.56016 (seconds)
Total = 7.59489 (seconds) 我的硬件和电路板支持包配置： SOC ：iMX Verdin EVK SoM V1.0C 载板：iMX Verdin EVK v1.2A（ https://www.toradex.com/de/computer-on-modules/verdin-arm-family/nxp-imx95-evaluation-kit?srsltid=AfmBOookfBHOVzIUBJhrVmtYSV45ohtPJy_f2ymYGR956gzD0XzxS0jP ） 我使用的版本是： MACHINE = "imx95-19x19-verdin" 我使用的是这个BSP版本： https://www.toradex.com/de/news/bsp-layers-reference-images-walnascar?srsltid =AfmBOoqJF2YHULiIe7sA1L4gPBasmZrepuGIRBLrtcIuFPnz39jctBX6 以及 NXP 各层： meta-imx rel_imx_6.12.49_2.2.0 内核版本： 6.12.49-lts-next-g759f4038100f 我的主要问题是，我的芯片版本（ A0）是否受支持，或者为什么运行模型时会静默失败并只产生乱码输出： 人脸检测（UltraFace-Ultraslim， [1,128,128,3] int8/uint8 输入）输出一个融合的(172,6)张量，每个锚点包含[bg_score, face_score, xmin, ymin, xmax, ymax] ，已经过 NMS 处理并归一化到[0,1] — 在 CPU 上，这会产生一个清晰的高置信度检测结果 (~0.996)，紧密地包围着人脸，而在 NPU 上，所有 172 个锚点都坍缩成一个相同的常数 (~0.50 分，接近零大小的框位于 ~0.227,0.227,0.227,0.227)。面部特征点（NXP facial_landmarks_35 ， [1,60,60,3] uint8 输入）输出一个[1,70]张量，包含 35 个交错的 (x,y) 点，并归一化到面部裁剪区域——在 CPU 上，当叠加到图像上时，这些点会形成一个可识别的面部特征点模式；而在 NPU 上，整个 70 个值的输出同样会坍缩成一个重复的常量，而不是每个点都不同。 Yocto Project Re: NPU support on the i.MX95 Verdin EVK 输出详细信息： root@imx95-19x19-verdin-4798be6ce85542d2:/usr/bin/tensorflow-lite-2.19.0/examples# ./label_image -m demo_converted -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libneutron_delegate.so -v 1 -r 5
INFO: Loaded model demo_converted
INFO: resolved reporter
INFO: tensors size: 11
INFO: nodes size: 4
INFO: inputs: 1
INFO: input(0) name: input
INFO: 0: MobilenetV1/Logits/SpatialSqueeze, 1001, 9, 0.166099, -62
INFO: 1: MobilenetV1/Predictions/Reshape_1, 1001, 3, 0.00390625, 0
INFO: 2: input, 150528, 3, 0.0078125, 128
INFO: 3: MobilenetV1/Predictions/Reshape_1/requantize, 1001, 9, 0.00390625, -128
INFO: 4: input, 150528, 9, 0.0078125, 0
INFO: 5: NeutronMicrocode, 23944, 3, 0, 0
INFO: 6: NeutronWeights, 4329648, 3, 0, 0
INFO: 7: NeutronKernels, 11088, 3, 0, 0
INFO: 8: NeutronScratch, 380912, 3, 0, 0
INFO: 9: NeutronProfile, 0, 3, 0, 0
INFO: 10: NeutronDebug, 0, 3, 0, 0
INFO: len: 940650
INFO: width, height, channels: 517, 606, 3
INFO: input: 2
INFO: number of inputs: 1
INFO: number of outputs: 1
INFO: EXTERNAL delegate created.
INFO: NeutronDelegate delegate: 1 nodes delegated out of 4 nodes with 1 partitions.
INFO: Neutron delegate version: v1.0.0-f24d08e5, zerocp enabled.
INFO: Applied EXTERNAL delegate.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Interpreter has 1 subgraphs.
-----------Subgraph-0 has 11 tensors and 5 nodes------------
1 Inputs: [2] -> 150528B (0.14MB)
1 Outputs: [1] -> 1001B (0.00MB)
Tensor ID Name Type AllocType Size (Bytes/MB) Shape MemAddr-Offset
Tensor 0 MobilenetV1/Logits/Spa... kTfLiteInt8 kTfLiteCustom 1001 / 0.00 [1,1001] [-1, -1)
Tensor 1 MobilenetV1/Prediction... kTfLiteUInt8 kTfLiteArenaRw 1001 / 0.00 [1,1001] [151552, 152553)
Tensor 2 input kTfLiteUInt8 kTfLiteArenaRw 150528 / 0.14 [1,224,224,3] [0, 150528)
Tensor 3 MobilenetV1/Prediction... kTfLiteInt8 kTfLiteArenaRw 1001 / 0.00 [1,1001] [150528, 151529)
Tensor 4 input kTfLiteInt8 kTfLiteCustom 150528 / 0.14 [1,224,224,3] [-1, -1)
Tensor 5 NeutronMicrocode kTfLiteUInt8 kTfLiteMmapRo 23944 / 0.02 [23944] [4340768, 4364712)
Tensor 6 NeutronWeights kTfLiteUInt8 kTfLiteMmapRo 4329648 / 4.13 [4329648] [11104, 4340752)
Tensor 7 NeutronKernels kTfLiteUInt8 kTfLiteMmapRo 11088 / 0.01 [11088] [0, 11088)
Tensor 8 NeutronScratch kTfLiteUInt8 kTfLiteArenaRw 380912 / 0.36 [380912] [-1, -1)
Tensor 9 NeutronProfile kTfLiteUInt8 kTfLiteArenaRw 0 / 0.00 [0] [-1, -1)
Tensor 10 NeutronDebug kTfLiteUInt8 kTfLiteArenaRw 0 / 0.00 [0] [-1, -1)
kTfLiteArenaRw Info:
Tensor 2 has the max size 150528 bytes (0.144 MB).
This memory arena is estimated as[0xaaaafd73ffa9, 0xaaaafd71abc0), taking 152553 bytes (0.145 MB).
One possible set of tensors that have non-overlapping memory spaces with each other, and they take up the whole arena:
Tensor 2 -> 3 -> 1.
kTfLiteArenaRwPersistent Info: not holding any allocation.
kTfLiteMmapRo Info:
Tensor 6 has the max size 4329648 bytes (4.129 MB).
This memory arena is estimated as[0xffff7ec299f8, 0xffff7e800050), taking 4364712 bytes (4.163 MB).
One possible set of tensors that have non-overlapping memory spaces with each other, and they take up the whole arena:
Tensor 7 -> 6 -> 5.
kTfLiteDynamic Info: not holding any allocation.
=== Beginning of kTfLiteArenaRw Dump: ===
Total size is 152553 bytes (0.145 MB), holding 3 tensors.
tensor 2: life_span: node [0, 4], size: 150528 bytes (0.144 MB).
tensor 3: life_span: node [2, 3], size: 1001 bytes (0.001 MB).
tensor 1: life_span: node [3, 4], size: 1001 bytes (0.001 MB).
1 tensors are of same max size (150528 B (0.144 MB)): [2]
Per-layer-info in the order of op execution:
Node 0: 150528 bytes (0.144 MB), utilization rate: 98.673%, 1 live tensors: [2]
Node 4: 151529 bytes (0.145 MB), utilization rate: 99.329%, 2 live tensors: [1,2]
Node 2: 151529 bytes (0.145 MB), utilization rate: 99.329%, 2 live tensors: [2,3]
Node 3: 152530 bytes (0.145 MB), utilization rate: 99.985%, 3 live tensors: [1-3]
Top 4 memory-consuming layers:
Node 3: 152530 bytes (0.145 MB), utilization rate: 99.985%, 3 live tensors: [1-3]
Node 4: 151529 bytes (0.145 MB), utilization rate: 99.329%, 2 live tensors: [1,2]
Node 2: 151529 bytes (0.145 MB), utilization rate: 99.329%, 2 live tensors: [2,3]
Node 0: 150528 bytes (0.144 MB), utilization rate: 98.673%, 1 live tensors: [2]
===End of kTfLiteArenaRw Dump: ===
Node 0 Operator Builtin Code 114 QUANTIZE (not delegated)
1 Input Tensors:[2] -> 150528B (0.14MB)
1 Output Tensors:[4] -> 150528B (0.14MB)
Node 1 Operator Custom Name NeutronGraph (delegated by node 4)
4 Input Tensors:[4,5,6,7] -> 0B (0.00MB)
4 Output Tensors:[0,8-10] -> 0B (0.00MB)
Node 2 Operator Builtin Code 25 SOFTMAX (not delegated)
1 Input Tensors:[0] -> 1001B (0.00MB)
1 Output Tensors:[3] -> 1001B (0.00MB)
Node 3 Operator Builtin Code 114 QUANTIZE (not delegated)
1 Input Tensors:[3] -> 1001B (0.00MB)
1 Output Tensors:[1] -> 1001B (0.00MB)
Node 4 Operator Custom Name NeutronDelegate
4 Input Tensors:[4-7] -> 4515208B (4.31MB)
1 Output Tensors:[0] -> 1001B (0.00MB)
Execution plan as the list of 4 nodes invoked in-order: [0,4,2,3]
Among these nodes in the execution plan:
Node 4 is a NeutronDelegate node (0xaaaafd6f5c30), which has delegated 1 nodes: [1]
--------------Subgraph-0 dump has completed--------------
--------------Memory Arena Status Start--------------
Total memory usage: 152553 bytes (0.145 MB)
- Total arena memory usage: 152553 bytes (0.145 MB)
- Total dynamic memory usage: 0 bytes (0.000 MB)
Subgraph#0 Arena (Normal) 152553 (100.00%)
--------------Memory Arena Status End--------------
INFO: invoked
INFO: average time: 0.326 ms
查看全文
