2397295_en-US

[#1]

Performance degradation in Facemesh Landmark model ptq conversion

We were using Google's FaceMesh model released by NXP after ptq in this repo:
nxp-demo-experience-demos-list/downloads.json at lf-6.12.3_1.0.0 · nxp-imx-support/nxp-demo-experien...
But this model is based on Google's old FaceMesh model, which had 468 landmark points.

Now we want to move to Google's new FaceMesh model, which has 478 landmark points. We want to run this on iMX 95 FRDM board NPU. So, we wanted to quantize this. We are using NXP's eIQ-neutron-sdk-linux-3.1.3 to quantize this model.

After this quantization, we are seeing significant degradation in model performance, almost unusable for actually using it.

Initially we were quantizing it with MIN-MAX option. That model was unusable. Then we tried using percentile option and found a better performance with percentile set to 95 (even though this was regressor outputs).
But this is still not giving great performance

1) When NXP created the ptq file for old FaceMesh(468) model which option did they use, MIN-MAX? Or percentile?

2) Is there anything else we need to check when the performance degrade drastically after quantization

3) we profiled with the CelebA dataset used the serialize_image.py in scripts dir with model options specific options, should we run the full media pipe and create the calibration dataset or make changes in serialize script 

options supplied

serialize_image.py: 
-i            //218 x 178 front facing RGB images 
-o
-f bin
-t float32
-m 0to1
-s 256, 256
-layout NHWC
-co RGB

tflite-profiler:
--input 
--dataset
--output

tflite-quantizer:
--input 
--profile 
--quantize-inputs=false     
--quantize-outputs=false
--quantization-calibration-method= //MinMax or Percentile

Re: Performance degradation in Facemesh Landmark model ptq conversion

Hi @dhilshad,
Thank you for contacting NXP Support!

1) Unfortunately, we do not have that information available at this time.

2) This behavior is expected. Quantization is only one part of the deployment process; converting and optimizing a model for execution on embedded hardware involves several additional steps, such as graph optimization, operator mapping, hardware-specific transformations, and runtime validation. As a result, model behavior and performance can vary even when the model is already quantized.

For new designs and evaluations, I recommend using eIQ Olive, as it provides a more modern framework for model optimization and deployment on NXP i.MX platforms. It includes updated workflows and improved support for current machine learning deployment scenarios.

Please refer to the following tutorials and documentation for more information:

https://eiq.nxp.com/learning-hub/tools/olive/index.html

These resources cover the recommended workflows and best practices for deploying machine learning models on i.MX devices.

Best regards,
Chavira

Re: Performance degradation in Facemesh Landmark model ptq conversion

Hi @Chavira , Thanks for the replay. And pointing out to the documentation

Just to clarify here, our main concern is the accuracy of the model . In case of FaceMesh model, we see that the output landmark points are not accurate enough for our application. As I said earlier, we had found that keeping a 95 percentile cutoff was giving a slightly better result than the MIN MAX. But still not comparable to the accuracy we see in the NXP's ptq model (FaceMesh 468). 

We have one more observation:
1) We tried by giving just 8 samples from our production environment as representative dataset. This was slightly improving the result. Then we added 250 images from the same environment and profiled and quantized. But this caused the accuracy to degrade. 
Do you have any idea why this kind of behaviour might come?

2) Also, has NXP already converted the Google's new FaceMesh model (with 478 landmark) model to ptq?