IMX8MP NPU produce same output with same input

RIMASTA · ‎01-21-2025

IMX8MP produce same output with same input. Same as

tensorflow-imx at branch lf-5.10.52_2.1.0,

Dynamic link to:
libusb-1.0.so.0

libtensorflow-lite.so.2.9.1

libopencv_core.so.406

libopencv_video.so.406
libopencv.....
Gstreamer is from NXP yocto build, Initializing GStreamer Core Library version 1.20.3

in words:
Here are use cases.

Case 1:

I open camera using LIBUSB, and then close it (I do not read anything just, claim and unclaim interface).
I open camera with opencv gstreamer backend, after NPU initialization -> I get the same output with different input from NPU.

Case 2:

1.I DO NOT open+close with LIBUSB.

2.I open camera with opencv gstreamer backend after NPU initialization -> WORKS!

Case 3:

I open camera using LIBUSB, and then close it (I do not read anything, just claim and unclaim interface).
I open camera with opencv V4l2 backend after NPU initialization -> WORKS!

Questions:

How can opencv+gstreamer change output from NPU, I only use cpu plugins in gst?
Can you review the off init() USBCameraDevice?
Can you review off initialization off NPU, or provide a working example?

Same bug is reported here:

https://community.nxp.com/t5/i-MX-Processors/tflite-runtime-NNAPI-stops-working-when-a-computational...

With some differences:
I use C++ and vxdelegate.

I cannot reproduce his result, I must do the LIBUSB init().

Code:

bool init()

{

if (libusb_init(&context_) != 0)

{

std::cout << "libusb_init failed " << std::endl;

return false;

}

handle_ = libusb_open_device_with_vid_pid(context_, 0x0BDA, 0x5880);

if (handle_ == nullptr)

{

std::cout << "libusb_open_device_with_vid_pid failed " << std::endl;

return false;

}

kernel_driver_attached_ = (libusb_kernel_driver_active(handle_, 0) == 1);

if (kernel_driver_attached_)

{

if (libusb_detach_kernel_driver(handle_, 0) != 0)

{

std::cout << "libusb_detach_kernel_driver failed " << std::endl;

return false;

}

if (libusb_claim_interface(handle_, 0) != 0)

{

std::cout << "libusb_claim_interface " << std::endl;

return false;

}

init_ok_ = true;

return true;

}

#include "bug.h"

#include "tensorflow/lite/delegates/external/external_delegate.h"

#include "tensorflow/lite/interpreter.h"

#include "tensorflow/lite/kernels/register.h"

#include "tensorflow/lite/model.h"

#include "tensorflow/lite/optional_debug_tools.h"

#include "tensorflow/lite/tools/delegates/delegate_provider.h"

#include "USB_camera_device.hpp"

#include <chrono>

#include <ctime>

#include <iostream>

#include <numeric>

CameraNode::CameraNode() : running_(false)

{

USBCameraDevice camera_interface;

bool camera_interface_ok = camera_interface.init(); // comment this no bug!

}

CameraNode::~CameraNode()

{

running_ = false;

}

void CameraNode::start()

{

running_ = true;

std::thread t1(&CameraNode::captureThread, this);

std::thread t2(&CameraNode::processingThread, this);

t1.join();

t2.join();

}

void CameraNode::captureThread()

{

auto gstreamerPipeline =

"v4l2src device=/dev/videoFRONT ! video/x-raw,format=YUY2,width=812,height=420,framerate=25/1 ! appsink";

std::this_thread::sleep_for(std::chrono::milliseconds(20000));

std::cout << "cam open " << std::endl;

// cv::VideoCapture cap("/dev/videoFRONT", cv::CAP_V4L2); // CAP_V4L2 no bug, bug with gstreamer!

// if (!cap.isOpened())

// {

// std::cerr << "Error: Unable to open camera at" << std::endl;

// return;

// }

// // Set camera properties (optional, but recommended)

// // Set the desired properties for YUY2 format

// cap.set(cv::CAP_PROP_FRAME_WIDTH, 812);

// cap.set(cv::CAP_PROP_FRAME_HEIGHT, 420);

// cap.set(cv::CAP_PROP_FPS, 25);

// cap.set(cv::CAP_PROP_FOURCC, cv::VideoWriter::fourcc('Y', 'U', 'Y', 'V')); // YUY2 format

// cap.set(cv::CAP_PROP_CONVERT_RGB, 0);

cv::VideoCapture cap;

cap.open(gstreamerPipeline, cv::CAP_GSTREAMER); // <- bug

std::cout << "done open " << std::endl;

if (!cap.isOpened())

{

std::cerr << "Error: Unable to open the camera using GStreamer pipeline." << std::endl;

running_ = false;

return;

}

std::cout << "Camera opened successfully." << std::endl;

cv::Mat frame;

while (running_)

{

cap >> frame;

std::cout << "cap cnls: " << frame.channels() << " w " << frame.cols << " " << frame.rows << std::endl;

if (frame.empty())

{

std::cerr << "Error: Received empty frame." << std::endl;

running_ = false;

break;

}

cap.release();

}

void CameraNode::processingThread()

{

// #################REVIEW START#############################

std::unique_ptr<tflite::Interpreter> npu_interpreter_cpu_fallback;

std::string model_path_ =

"/usr/share/camera/models/seg_14x14_752_192.tflite"; // ANY MODEL FAILS INCLUDING MOBILENET from tensorflow

// examples

std::unique_ptr<tflite::FlatBufferModel> model_ = tflite::FlatBufferModel::BuildFromFile(model_path_.c_str());

tflite::ops::builtin::BuiltinOpResolver resolver_;

tflite::InterpreterBuilder builder(*model_, resolver_);

builder(&npu_interpreter_cpu_fallback);

std::string delegate_string_ = "/usr/lib/libvx_delegate.so";

auto delegate_options_ = TfLiteExternalDelegateOptionsDefault(delegate_string_.c_str());

auto delegate_ptr_ = TfLiteExternalDelegateCreate(&delegate_options_);

auto ok = npu_interpreter_cpu_fallback->ModifyGraphWithDelegate(delegate_ptr_);

std::cout << "allocate " << std::endl;

if (npu_interpreter_cpu_fallback->AllocateTensors() != kTfLiteOk)

{

std::cout << "npu ok " << std::endl;

}

// #################REVIEW END#############################

while (running_)

{

auto in_tensor = npu_interpreter_cpu_fallback->input_tensor(0);

auto input_width = in_tensor->dims->data[2];

auto input_height = in_tensor->dims->data[1];

auto input_channels = 3;

cv::Mat randomImage(input_height, input_width, CV_8UC3);

cv::randu(randomImage, cv::Scalar(0, 0, 0), cv::Scalar(255, 255, 255));

int out_sum = 0;

int in_sum = 0;

// Retrieve the latest frame safely

memcpy(npu_interpreter_cpu_fallback->typed_input_tensor<uint8_t>(0),

randomImage.data,

input_width * input_height * input_channels * sizeof(uint8_t));

for (size_t i = 0; i < input_channels * input_width * input_height; i++)

{

in_sum += npu_interpreter_cpu_fallback->typed_input_tensor<uint8_t>(0)[i];

}

std::cout << "invoke, input sum " << in_sum << std::endl;

auto inference_ok = npu_interpreter_cpu_fallback->Invoke();

cv::Mat output;

// std::cout << " SIZE " << npu_interpreter_cpu_fallback->output_tensor(0)->dims->size << std::endl;

auto output_height_ = npu_interpreter_cpu_fallback->output_tensor(0)->dims->data[1];

int output_channels_ = 1;

auto output_width_ = 1;

if (npu_interpreter_cpu_fallback->output_tensor(0)->dims->size > 2)

{

auto output_width_ = npu_interpreter_cpu_fallback->output_tensor(0)->dims->data[2];

output_channels_ = npu_interpreter_cpu_fallback->output_tensor(0)->dims->data[3];

}

output.create(output_height_, output_width_, CV_8UC(output_channels_));

switch (npu_interpreter_cpu_fallback->output_tensor(0)->type)

{

case kTfLiteUInt8:

{

int s = output_channels_ * output_height_ * output_width_;

memcpy(output.data,

npu_interpreter_cpu_fallback->typed_output_tensor<uint8_t>(0),

output_channels_ * output_height_ * output_width_ * sizeof(uint8_t));

for (size_t i = 0; i < s; i++)

{

out_sum += int(npu_interpreter_cpu_fallback->typed_output_tensor<uint8_t>(0)[i]);

}

break;

}

case kTfLiteInt8:

case kTfLiteFloat32:

case kTfLiteInt16:

case kTfLiteInt32:

case kTfLiteBool:

case kTfLiteNoType:

case kTfLiteFloat64:

case kTfLiteInt64:

case kTfLiteString:

case kTfLiteComplex64:

case kTfLiteComplex128:

case kTfLiteFloat16:

case kTfLiteUInt64:

case kTfLiteResource:

case kTfLiteVariant:

case kTfLiteUInt32:

case kTfLiteUInt16:

default:

{

break;

}

if (!output.empty())

{

std::cout << "Sum of pixels in the latest frame 2: " << out_sum << std::endl;

std::this_thread::sleep_for(std::chrono::milliseconds(2));

}

#ifndef CAMERANODE_H

#define CAMERANODE_H

#include <atomic>

#include <ctime>

#include <mutex>

#include <opencv2/opencv.hpp>

#include <thread>

class CameraNode

{

public:

CameraNode();

~CameraNode();

void start();

private:

void captureThread();

void processingThread();

std::atomic<bool> running_;

};

#endif // CAMERANODE_H

Log from gstreamer:

0:00:00.004675375 94448 0xffff781cfe00 INFO GST_INIT gst.c:592:init_pre: Initializing GStreamer Core Library version 1.20.3

0:00:00.005013125 94448 0xffff781cfe00 INFO GST_INIT gst.c:593:init_pre: Using library installed in /usr/lib

0:00:00.005216250 94448 0xffff781cfe00 INFO GST_INIT gst.c:611:init_pre: Linux occulus93f0 5.15.52-5.15.87-2.2.0-greenworks+g8b2bcc215bd8+p10 #1 SMP PREEMPT Thu Feb 2 07:57:31 UTC 2023 aarch64

0:00:00.007921375 94448 0xffff781cfe00 INFO GST_INIT gstmessage.c:129:_priv_gst_message_initialize: init messages

0:00:00.015012375 94448 0xffff781cfe00 INFO GST_INIT gstcontext.c:86:_priv_gst_context_initialize: init contexts

0:00:00.016300250 94448 0xffff781cfe00 INFO GST_PLUGIN_LOADING gstplugin.c:324:_priv_gst_plugin_initialize: registering 0 static plugins

0:00:00.016978250 94448 0xffff781cfe00 INFO GST_PLUGIN_LOADING gstplugin.c:232:gst_plugin_register_static: registered static plugin "staticelements"

0:00:00.017076000 94448 0xffff781cfe00 INFO GST_PLUGIN_LOADING gstplugin.c:234:gst_plugin_register_static: added static plugin "staticelements", result: 1

0:00:00.017446875 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistry.c:1827:ensure_current_registry: reading registry cache: /home/root/.cache/gstreamer-1.0/registry.aarch64.bin

Sum of pixels in the latest frame 2: 12241

0:00:00.065254875 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistrybinary.c:683:priv_gst_registry_binary_read_cache: loaded /home/root/.cache/gstreamer-1.0/registry.aarch64.bin in 0.047598 seconds

0:00:00.065506625 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistry.c:1693:scan_and_update_registry: Validating plugins from registry cache: /home/root/.cache/gstreamer-1.0/registry.aarch64.bin

invoke, input sum 55062968

0:00:00.079203875 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistry.c:1375:gst_registry_scan_path_level:<registry0> cached info for /usr/lib/gstreamer-1.0/libgstvideo4linux2.so is stale

Sum of pixels in the latest frame 2: 12251

0:00:00.000165000 94476 0xaaaacbe10400 INFO GST_INIT gst.c:592:init_pre: Initializing GStreamer Core Library version 1.20.3

0:00:00.000265125 94476 0xaaaacbe10400 INFO GST_INIT gst.c:593:init_pre: Using library installed in /usr/lib

0:00:00.000294125 94476 0xaaaacbe10400 INFO GST_INIT gst.c:611:init_pre: Linux occulus93f0 5.15.52-5.15.87-2.2.0-greenworks+g8b2bcc215bd8+p10 #1 SMP PREEMPT Thu Feb 2 07:57:31 UTC 2023 aarch64

0:00:00.000782375 94476 0xaaaacbe10400 INFO GST_INIT gstmessage.c:129:_priv_gst_message_initialize: init messages

0:00:00.002059375 94476 0xaaaacbe10400 INFO GST_INIT gstcontext.c:86:_priv_gst_context_initialize: init contexts

0:00:00.002547375 94476 0xaaaacbe10400 INFO GST_PLUGIN_LOADING gstplugin.c:324:_priv_gst_plugin_initialize: registering 0 static plugins

0:00:00.002815500 94476 0xaaaacbe10400 INFO GST_PLUGIN_LOADING gstplugin.c:232:gst_plugin_register_static: registered static plugin "staticelements"

0:00:00.002855000 94476 0xaaaacbe10400 INFO GST_PLUGIN_LOADING gstplugin.c:234:gst_plugin_register_static: added static plugin "staticelements", result: 1

0:00:00.002962000 94476 0xaaaacbe10400 INFO GST_REGISTRY gstregistry.c:1862:ensure_current_registry: registry reading and updating done, result = 1

0:00:00.002991875 94476 0xaaaacbe10400 INFO GST_INIT gst.c:833:init_post: GLib runtime version: 2.72.3

0:00:00.003016750 94476 0xaaaacbe10400 INFO GST_INIT gst.c:835:init_post: GLib headers version: 2.72.3

0:00:00.003037250 94476 0xaaaacbe10400 INFO GST_INIT gst.c:837:init_post: initialized GStreamer successfully

invoke, input sum 55069542

Sum of pixels in the latest frame 2: 12251

0:00:00.065827250 94476 0xaaaacbe10400 INFO GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/gstreamer-1.0/libgstvideo4linux2.so" loaded

0:00:00.223206250 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistry.c:1794:scan_and_update_registry: Registry cache changed. Writing new registry cache

0:00:00.223331125 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistrybinary.c:431:priv_gst_registry_binary_write_cache: Building binary registry cache image

0:00:00.244991125 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistrybinary.c:463:priv_gst_registry_binary_write_cache: Writing binary registry cache

invoke, input sum 55112192

0:00:00.278634500 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistrybinary.c:314:gst_registry_binary_cache_finish: Wrote binary registry cache

0:00:00.278768250 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistry.c:1803:scan_and_update_registry: Registry cache written successfully

0:00:00.278894125 94448 0xffff781cfe00 INFO GST_REGISTRY gstregistry.c:1862:ensure_current_registry: registry reading and updating done, result = 1

0:00:00.279017375 94448 0xffff781cfe00 INFO GST_INIT gst.c:833:init_post: GLib runtime version: 2.72.3

0:00:00.279165625 94448 0xffff781cfe00 INFO GST_INIT gst.c:835:init_post: GLib headers version: 2.72.3

0:00:00.279259125 94448 0xffff781cfe00 INFO GST_INIT gst.c:837:init_post: initialized GStreamer successfully

0:00:00.280262125 94448 0xffff781cfe00 INFO GST_PIPELINE gstparse.c:344:gst_parse_launch_full: parsing pipeline description 'v4l2src device=/dev/videoFRONT ! video/x-raw,format=YUY2,width=812,height=420,framerate=25/1 ! appsink'

Sum of pixels in the latest frame 2: 12251

invoke, input sum 55048696

0:00:00.333112125 94448 0xffff781cfe00 INFO GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/gstreamer-1.0/libgstvideo4linux2.so" loaded

0:00:00.333303750 94448 0xffff781cfe00 INFO GST_ELEMENT_FACTORY gstelementfactory.c:489:gst_element_factory_create_with_properties: creating element "v4l2src"

0:00:00.356930500 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstelement.c:759:gst_element_add_pad:<GstBaseSrc@0xffff793ec160> adding pad 'src'

0:00:00.363878250 94448 0xffff781cfe00 INFO GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/gstreamer-1.0/libgstapp.so" loaded

0:00:00.364000875 94448 0xffff781cfe00 INFO GST_ELEMENT_FACTORY gstelementfactory.c:489:gst_element_factory_create_with_properties: creating element "appsink"

0:00:00.364985000 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstelement.c:759:gst_element_add_pad:<GstBaseSink@0xffff62957c90> adding pad 'sink'

0:00:00.365291500 94448 0xffff781cfe00 INFO GST_ELEMENT_FACTORY gstelementfactory.c:489:gst_element_factory_create_with_properties: creating element "pipeline"

0:00:00.366244500 94448 0xffff781cfe00 INFO GST_PIPELINE gst/parse/grammar.y:683:gst_parse_perform_link: linking some pad of GstV4l2Src named v4l2src0 to some pad of GstAppSink named appsink0 (0/0) with caps "video/x-raw, format=(string)YUY2, width=(int)812, height=(int)420, framerate=(fraction)25/1"

Sum of pixels in the latest frame 2: 12251

0:00:00.376754500 94448 0xffff781cfe00 INFO GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/gstreamer-1.0/libgstcoreelements.so" loaded

0:00:00.377027125 94448 0xffff781cfe00 INFO GST_ELEMENT_FACTORY gstelementfactory.c:489:gst_element_factory_create_with_properties: creating element "capsfilter"

0:00:00.377737125 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstelement.c:759:gst_element_add_pad:<GstBaseTransform@0xffff793fc100> adding pad 'sink'

0:00:00.377952000 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstelement.c:759:gst_element_add_pad:<GstBaseTransform@0xffff793fc100> adding pad 'src'

0:00:00.378164875 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2069:gst_bin_get_state_func:<pipeline0> getting state

0:00:00.378436000 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<capsfilter0> completed state change to NULL

0:00:00.378571875 94448 0xffff781cfe00 INFO GST_EVENT gstevent.c:1660:gst_event_new_reconfigure: creating reconfigure event

0:00:00.378761875 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstutils.c:1816:gst_element_link_pads_full: trying to link element v4l2src0:(any) to element capsfilter0:sink

0:00:00.378881375 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstelement.c:1016:gst_element_get_static_pad: found pad capsfilter0:sink

0:00:00.379019375 94448 0xffff781cfe00 INFO GST_PADS gstutils.c:1632:prepare_link_maybe_ghosting: v4l2src0 and capsfilter0 in same bin, no need for ghost pads

0:00:00.379178750 94448 0xffff781cfe00 INFO GST_PADS gstpad.c:2382:gst_pad_link_prepare: trying to link v4l2src0:src and capsfilter0:sink

0:00:00.379337750 94448 0xffff781cfe00 INFO GST_PADS gstpad.c:4357:gst_pad_peer_query:<capsfilter0:src> pad has no peer

0:00:00.379531625 94448 0xffff781cfe00 INFO GST_PADS gstpad.c:2590:gst_pad_link_full: linked v4l2src0:src and capsfilter0:sink, successful

0:00:00.379611500 94448 0xffff781cfe00 INFO GST_EVENT gstevent.c:1660:gst_event_new_reconfigure: creating reconfigure event

0:00:00.379685500 94448 0xffff781cfe00 INFO GST_EVENT gstpad.c:5946:gst_pad_send_event_unchecked:<v4l2src0:src> Received event on flushing pad. Discarding

0:00:00.379844000 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstutils.c:1816:gst_element_link_pads_full: trying to link element capsfilter0:src to element appsink0:(any)

0:00:00.379952625 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstelement.c:1016:gst_element_get_static_pad: found pad capsfilter0:src

0:00:00.380072125 94448 0xffff781cfe00 INFO GST_PADS gstutils.c:1079:gst_pad_check_link: trying to link capsfilter0:src and appsink0:sink

0:00:00.380371125 94448 0xffff781cfe00 INFO GST_PADS gstutils.c:1632:prepare_link_maybe_ghosting: capsfilter0 and appsink0 in same bin, no need for ghost pads

0:00:00.380522250 94448 0xffff781cfe00 INFO GST_PADS gstpad.c:2382:gst_pad_link_prepare: trying to link capsfilter0:src and appsink0:sink

0:00:00.380725250 94448 0xffff781cfe00 INFO GST_PADS gstpad.c:2590:gst_pad_link_full: linked capsfilter0:src and appsink0:sink, successful

0:00:00.380802125 94448 0xffff781cfe00 INFO GST_EVENT gstevent.c:1660:gst_event_new_reconfigure: creating reconfigure event

0:00:00.380887375 94448 0xffff781cfe00 INFO GST_EVENT gstpad.c:5946:gst_pad_send_event_unchecked:<capsfilter0:src> Received event on flushing pad. Discarding

0:00:00.381162250 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstelement.c:1016:gst_element_get_static_pad: found pad appsink0:sink

0:00:00.381603500 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<appsink0> current NULL pending VOID_PENDING, desired next READY

0:00:00.381721375 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<appsink0> completed state change to READY

0:00:00.381846125 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<appsink0> notifying about state-changed NULL to READY (VOID_PENDING pending)

0:00:00.382106625 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline0> child 'appsink0' changed state to 2(READY) successfully

0:00:00.382227125 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<capsfilter0> current NULL pending VOID_PENDING, desired next READY

0:00:00.382312500 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<capsfilter0> completed state change to READY

0:00:00.382398000 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<capsfilter0> notifying about state-changed NULL to READY (VOID_PENDING pending)

0:00:00.382505125 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline0> child 'capsfilter0' changed state to 2(READY) successfully

0:00:00.382599000 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<v4l2src0> current NULL pending VOID_PENDING, desired next READY

invoke, input sum 54961436

Sum of pixels in the latest frame 2: 12251

invoke, input sum 55065912

0:00:00.515320375 94448 0xffff781cfe00 INFO v4l2 v4l2_calls.c:588:gst_v4l2_open:<v4l2src0:src> Opened device 'XYGD Camera: XYGD Camera' (/dev/videoFRONT) successfully

0:00:00.515769125 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<v4l2src0> completed state change to READY

0:00:00.515896500 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<v4l2src0> notifying about state-changed NULL to READY (VOID_PENDING pending)

0:00:00.516104000 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline0> child 'v4l2src0' changed state to 2(READY) successfully

0:00:00.516472000 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2778:gst_element_continue_state:<pipeline0> committing state from NULL to READY, pending PLAYING, next PAUSED

0:00:00.516824250 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<pipeline0> notifying about state-changed NULL to READY (PLAYING pending)

0:00:00.517029375 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2786:gst_element_continue_state:<pipeline0> continue state change READY to PAUSED, final PLAYING

0:00:00.517435000 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<appsink0> current READY pending VOID_PENDING, desired next PAUSED

0:00:00.517666750 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2935:gst_bin_change_state_func:<pipeline0> child 'appsink0' is changing state asynchronously to PAUSED

0:00:00.517890125 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<capsfilter0> current READY pending VOID_PENDING, desired next PAUSED

0:00:00.518086000 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<capsfilter0> completed state change to PAUSED

0:00:00.518341500 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<capsfilter0> notifying about state-changed READY to PAUSED (VOID_PENDING pending)

0:00:00.518657250 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline0> child 'capsfilter0' changed state to 3(PAUSED) successfully

0:00:00.518838750 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<v4l2src0> current READY pending VOID_PENDING, desired next PAUSED

0:00:00.519254000 94448 0xffff781cfe00 INFO basesrc gstbasesrc.c:1430:gst_base_src_do_seek:<v4l2src0> seeking: time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.000000000, duration 99:99:99.999999999

0:00:00.520841875 94448 0xffff781cfe00 INFO task gsttask.c:516:gst_task_set_lock: setting stream lock 0xffff793f2140 on task 0xffff50901050

0:00:00.521176500 94448 0xffff781cfe00 INFO GST_PADS gstpad.c:6291:gst_pad_start_task:<v4l2src0:src> created task 0xffff50901050

0:00:00.522258500 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<v4l2src0> completed state change to PAUSED

0:00:00.522535000 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<v4l2src0> notifying about state-changed READY to PAUSED (VOID_PENDING pending)

0:00:00.522623625 94448 0xffff619a4860 INFO GST_ELEMENT_PADS gstelement.c:1013:gst_element_get_static_pad: no such pad 'sink' in element "v4l2src0"

0:00:00.522858125 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2971:gst_bin_change_state_func:<pipeline0> child 'v4l2src0' changed state to 3(PAUSED) successfully without preroll

0:00:00.523078875 94448 0xffff781cfe00 INFO pipeline gstpipeline.c:533:gst_pipeline_change_state:<pipeline0> pipeline is live

0:00:00.523435000 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2778:gst_element_continue_state:<pipeline0> committing state from READY to PAUSED, pending PLAYING, next PLAYING

0:00:00.523500750 94448 0xffff619a4860 INFO v4l2src gstv4l2src.c:550:gst_v4l2src_query_preferred_size:<v4l2src0> Detect input 0 as `Camera 1`

0:00:00.523583250 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<pipeline0> notifying about state-changed READY to PAUSED (PLAYING pending)

0:00:00.523799875 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2786:gst_element_continue_state:<pipeline0> continue state change PAUSED to PLAYING, final PLAYING

0:00:00.523857625 94448 0xffff619a4860 INFO v4l2 gstv4l2object.c:1241:gst_v4l2_object_fill_format_list:<v4l2src0:src> got 2 format(s):

0:00:00.524240000 94448 0xffff619a4860 INFO v4l2 gstv4l2object.c:1245:gst_v4l2_object_fill_format_list:<v4l2src0:src> YUYV

0:00:00.524413000 94448 0xffff619a4860 INFO v4l2 gstv4l2object.c:1245:gst_v4l2_object_fill_format_list:<v4l2src0:src> MJPG

0:00:00.524410250 94448 0xffff781cfe00 INFO GST_EVENT gstevent.c:1530:gst_event_new_latency: creating latency event 0:00:00.000000000

0:00:00.524777375 94448 0xffff781cfe00 INFO bin gstbin.c:2759:gst_bin_do_latency_func:<pipeline0> configured latency of 0:00:00.000000000

0:00:00.525012000 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<appsink0> current READY pending PAUSED, desired next PLAYING

0:00:00.525226375 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2935:gst_bin_change_state_func:<pipeline0> child 'appsink0' is changing state asynchronously to PLAYING

0:00:00.525407500 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<capsfilter0> current PAUSED pending VOID_PENDING, desired next PLAYING

0:00:00.525527625 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<capsfilter0> completed state change to PLAYING

0:00:00.525656875 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<capsfilter0> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)

0:00:00.525884000 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline0> child 'capsfilter0' changed state to 4(PLAYING) successfully

0:00:00.526082125 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<v4l2src0> completed state change to PLAYING

0:00:00.526211750 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<v4l2src0> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)

0:00:00.526422125 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline0> child 'v4l2src0' changed state to 4(PLAYING) successfully

0:00:00.526642625 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2069:gst_bin_get_state_func:<pipeline0> getting state

0:00:00.526773125 94448 0xffff781cfe00 INFO GST_STATES gstelement.c:2522:gst_element_get_state_func:<pipeline0> waiting for element to commit state

0:00:00.527140250 94448 0xffff619a4860 INFO v4l2 gstv4l2object.c:4752:gst_v4l2_object_probe_caps:<v4l2src0:src> probed caps: video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)10/1; video/x-raw, format=(string)YUY2, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 20/1, 15/1, 10/1 }; video/x-raw, format=(string)YUY2, width=(int)720, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 25/1, 20/1, 15/1, 10/1 }; video/x-raw, format=(string)YUY2, width=(int)812, height=(int)420, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 25/1, 20/1, 15/1, 10/1 }; image/jpeg, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1, 25/1 }; image/jpeg, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1, 25/1 }; image/jpeg, width=(int)720, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1, 25/1 }; image/jpeg, width=(int)812, height=(int)420, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1, 25/1 }

Sum of pixels in the latest frame 2: 12251

0:00:00.547849875 94448 0xffff619a4860 INFO v4l2src gstv4l2src.c:647:gst_v4l2src_negotiate:<v4l2src0> fixated to: video/x-raw, format=(string)YUY2, width=(int)812, height=(int)420, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:5:1

0:00:00.548538875 94448 0xffff619a4860 INFO GST_EVENT gstevent.c:892:gst_event_new_caps: creating caps event video/x-raw, format=(string)YUY2, width=(int)812, height=(int)420, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:5:1

0:00:00.548959625 94448 0xffff619a4860 INFO basetransform gstbasetransform.c:1325:gst_base_transform_setcaps:<capsfilter0> reuse caps

0:00:00.549235750 94448 0xffff619a4860 INFO GST_EVENT gstevent.c:892:gst_event_new_caps: creating caps event video/x-raw, format=(string)YUY2, width=(int)812, height=(int)420, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:5:1

0:00:00.549558500 94448 0xffff619a4860 INFO GST_EVENT gstevent.c:892:gst_event_new_caps: creating caps event video/x-raw, format=(string)YUY2, width=(int)812, height=(int)420, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:5:1

invoke, input sum 55089027

0:00:00.587377125 94448 0xffff619a4860 INFO v4l2 gstv4l2object.c:4083:gst_v4l2_object_set_format_full:<v4l2src0:src> Set capture framerate to 25/1

0:00:00.587530875 94448 0xffff619a4860 INFO v4l2 gstv4l2object.c:3251:gst_v4l2_object_setup_pool:<v4l2src0:src> accessing buffers via mode 2

0:00:00.588640875 94448 0xffff619a4860 INFO v4l2bufferpool gstv4l2bufferpool.c:604:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool0:src> increasing minimum buffers to 2

0:00:00.588834500 94448 0xffff619a4860 INFO v4l2bufferpool gstv4l2bufferpool.c:617:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool0:src> reducing maximum buffers to 32

0:00:00.589517250 94448 0xffff619a4860 INFO v4l2bufferpool gstv4l2bufferpool.c:617:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool0:src> reducing maximum buffers to 32

0:00:00.592832375 94448 0xffff619a4860 WARN v4l2bufferpool gstv4l2bufferpool.c:884:gst_v4l2_buffer_pool_start:<v4l2src0:pool0:src> Uncertain or not enough buffers, enabling copy threshold

Sum of pixels in the latest frame 2: 12251

invoke, input sum 55026501

Sum of pixels in the latest frame 2: 12251

invoke, input sum 55091355

Sum of pixels in the latest frame 2: 12251

0:00:00.832698625 94448 0xffff619a4860 INFO GST_EVENT gstevent.c:972:gst_event_new_segment: creating segment event time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.000000000, duration 99:99:99.999999999

0:00:00.832894875 94448 0xffff619a4860 INFO basesrc gstbasesrc.c:3018:gst_base_src_loop:<v4l2src0> marking pending DISCONT

0:00:00.833110500 94448 0xffff619a4860 INFO GST_STATES gstbin.c:3406:bin_handle_async_done:<pipeline0> setting state from PAUSED to PAUSED, pending PLAYING

0:00:00.833233125 94448 0xffff619a4860 INFO GST_STATES gstbin.c:3428:bin_handle_async_done:<pipeline0> continue state change, pending PLAYING

0:00:00.833972375 94448 0xffff619a4060 INFO GST_STATES gstbin.c:3224:gst_bin_continue_func:<pipeline0> continue state change PAUSED to PLAYING, final PLAYING

0:00:00.834252250 94448 0xffff619a4060 INFO GST_EVENT gstevent.c:1530:gst_event_new_latency: creating latency event 0:00:00.060000000

0:00:00.834474000 94448 0xffff619a4060 INFO bin gstbin.c:2759:gst_bin_do_latency_func:<pipeline0> configured latency of 0:00:00.060000000

0:00:00.834652125 94448 0xffff619a4060 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<appsink0> current PAUSED pending VOID_PENDING, desired next PLAYING

0:00:00.834771250 94448 0xffff619a4060 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<appsink0> completed state change to PLAYING

0:00:00.834887000 94448 0xffff619a4060 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<appsink0> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)

invoke, input sum 55027005

0:00:00.835050875 94448 0xffff619a4060 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline0> child 'appsink0' changed state to 4(PLAYING) successfully

0:00:00.835177250 94448 0xffff619a4060 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<capsfilter0> current PLAYING pending VOID_PENDING, desired next PLAYING

0:00:00.835359125 94448 0xffff619a4060 INFO GST_STATES gstbin.c:2597:gst_bin_element_set_state:<capsfilter0> skipping transition from PLAYING to PLAYING

0:00:00.835518375 94448 0xffff619a4060 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline0> child 'capsfilter0' changed state to 4(PLAYING) successfully

0:00:00.835621375 94448 0xffff619a4060 INFO GST_STATES gstbin.c:2479:gst_bin_element_set_state:<v4l2src0> current PLAYING pending VOID_PENDING, desired next PLAYING

0:00:00.835725875 94448 0xffff619a4060 INFO GST_STATES gstbin.c:2597:gst_bin_element_set_state:<v4l2src0> skipping transition from PLAYING to PLAYING

0:00:00.835811375 94448 0xffff619a4060 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline0> child 'v4l2src0' changed state to 4(PLAYING) successfully

0:00:00.835946125 94448 0xffff619a4060 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<pipeline0> completed state change to PLAYING

0:00:00.836103750 94448 0xffff619a4060 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<pipeline0> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)

0:00:00.836392750 94448 0xffff781cfe00 INFO GST_ELEMENT_PADS gstelement.c:1016:gst_element_get_static_pad: found pad appsink0:sink

[ WARN:[email protected]] global /usr/src/debug/opencv/4.6.0.imx-r0/git/modules/videoio/src/cap_gstreamer.cpp (1405) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1

done open

Camera opened successfully.

0:00:00.837683125 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2069:gst_bin_get_state_func:<pipeline0> getting state

cap cnls: 2 w 812 420

0:00:00.840251500 94448 0xffff781cfe00 INFO GST_STATES gstbin.c:2069:gst_bin_get_state_func:<pipeline0> getting state

Sum of pixels in the latest frame 2: 12251

0:00:00.872442250 94448 0xffff619a4860 WARN v4l2src gstv4l2src.c:1123:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.311987500

Bio_TICFSL · ‎01-21-2025

Hello,

Please provide the code to tested since you are using gstreamer with npu and camera. Also you can check the machine learning guide:

https://www.nxp.com/docs/en/user-guide/IMX-MACHINE-LEARNING-UG.pdf

Regards

RIMASTA · ‎01-21-2025

I provide code. I do not know if you are building using yocto or sdk generated from yocto.
You will need to link to imx-tensorflow and opencv:

These are found in the imx-tensorflow branch lf-5.10.52_2.1.0, repo as described in the machine learning user guide:

add_executable(SOME_NAME path/bug.cpp

${TENSORFLOW_SOURCE_DIR}/tensorflow/lite/tools/delegates/default_execution_provider.cc

${TENSORFLOW_SOURCE_DIR}/tensorflow/lite/tools/delegates/delegate_provider.cc

${TENSORFLOW_SOURCE_DIR}/tensorflow/lite/tools/tool_params.cc

${TENSORFLOW_SOURCE_DIR}/tensorflow/lite/nnapi/sl/SupportLibrary.cc

${TENSORFLOW_SOURCE_DIR}/tensorflow/lite/nnapi/nnapi_implementation.cc

${TENSORFLOW_SOURCE_DIR}/tensorflow/lite/nnapi/nnapi_util.cc

${TENSORFLOW_SOURCE_DIR}/tensorflow/lite/tools/delegates/nnapi_delegate_provider.cc

${TENSORFLOW_SOURCE_DIR}/tensorflow/lite/tools/delegates/xnnpack_delegate_provider.cc

)

Bio_TICFSL · ‎01-22-2025

Hello,

Yes, it looks like the npu is not executed, will see with the developers thanks for the catch!

Regards

RIMASTA · ‎01-24-2025

Thanks, My workaround is v4l2, but Since root cause is not known I am forced to ship code to product with a potential bug. Anyway in case you find the root cause, please let me know even if you do not have a fix since I want more confidence that v4l2 is a good workaround.