topic Re: Gstreamer dropping frames (opencv videocapture) in i.MX Processors

Gstreamer dropping frames (opencv videocapture)

tyler_ward — Tue, 10 Mar 2020 14:31:19 GMT

I am using opencv to get fames from the CSI camera module on the IMX8M-mini dev kit. however every third frame is being dropped. Diging into the issue the issue is coming from the gstreamer backend and generates the filowing warnings when run with GST_DEBUG=2 .

0:00:14.053397814 4355 0x179c000 WARN v4l2src gstv4l2src.c:976:gst_v4l2src_create: lost frames detected: count = 1 - ts: 0:00:08.265484580

Testing in gstreamer directly the folowing commands show different results.

This looks to run fine

gst-launch-1.0 -v v4l2src device=/dev/video0 ! "video/x-raw, width=320, height=240, framerate=(fraction)30/1" ! fpsdisplaysink sync=false video-sink="fakesink"

This drops frames

gst-launch-1.0 -v v4l2src device=/dev/video0 ! "video/x-raw, width=320, height=240, framerate=(fraction)30/1" ! fpsdisplaysink sync=false video-sink="appsink"

Are other people having this issue as well? If so has anyone found a fix?

Re: Gstreamer dropping frames (opencv videocapture)

Bio_TICFSL — Thu, 12 Mar 2020 15:04:17 GMT

Hello Tyler,

Could you please, give me more details on the application? It will be important to know the BSP release too.

May the use of fakesink instead of appsink does not consume any data (I need to see the application to confirm it), so you will not be able to see the issue only by changing the sink plugin.

In any case, I will check if I can see this issue at some of our applications too.

Best Regards,

Re: Gstreamer dropping frames (opencv videocapture)

tyler_ward — Fri, 13 Mar 2020 12:14:59 GMT

The camera is being used for face detection and position tracking.

As well as in our application, the issue can be seen in a simple openCV application where the camera just captures frames in a loop. A mimimal example is shown below.

#include <cstdio>
#include <iostream>
#include <chrono>
#include <numeric>
#include <opencv2/core/mat.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/videoio.hpp>



int main(int argc, char* argv[])
{

    cv::VideoCapture cap(0);
    std::cout << cap.getBackendName() << std::endl;
    cap.set(cv::CAP_PROP_BUFFERSIZE, 2);
    cap.set(cv::CAP_PROP_FRAME_WIDTH, static_cast<double>(640) );
    cap.set(cv::CAP_PROP_FRAME_HEIGHT, static_cast<double>(480) );

    cv::Mat img;
    //perform a few initial captures to ensure correct setup
    for (int i = 0; i < 5; i++)
    {
        cap.read(img);
    }

    std::cout << "Camera framerate" << std::endl;

    //create time variables
    std::chrono::time_point<std::chrono::steady_clock> starttime;
    std::chrono::time_point<std::chrono::steady_clock> stoptime;

    starttime = std::chrono::steady_clock::now();
    for (int i = 0; i < 1000; i++)
    {
        cap.read(img);
    }
    stoptime = std::chrono::steady_clock::now();


    double time = ((double)std::chrono::duration_cast<std::chrono::milliseconds>(stoptime - starttime).count()) / 1000;
    double fps = 1000 / time;
    std::cout << "FPS = " << fps << "(" << time/200 << "s)" << std::endl;

}

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Our build is based arround the imx_4.14.98_2.0.0_ga manifest from the imx-linux-sumo branch.

The bsp is version: sumo-4.14.78-1.0.0_ga. The yocto status information for the nxp layers is shown below.

Build Configuration:
BB_VERSION           = "1.38.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-18.04"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "imx8mmevk"
DISTRO               = "fsl-imx-xwayland"
DISTRO_VERSION       = "4.14-sumo"
TUNE_FEATURES        = "aarch64"
TARGET_FPU           = ""
meta
meta-poky            = "HEAD:c9bd4984f8f471ca2c43052714f4413ba99cf171"
meta-oe
meta-multimedia      = "HEAD:8760facba1bceb299b3613b8955621ddaa3d4c3f"
meta-freescale       = "HEAD:27ca94f8a4336790ba117b4298566f6820e7e74c"
meta-freescale-3rdparty = "HEAD:82037216280a39957fb4272581637abec734ad50"
meta-freescale-distro = "HEAD:f7e2216e93aff14ac32728a13637a48df436b7f4"
meta-bsp
meta-sdk             = "sumo-4.14.78-1.0.0_ga:43dc308f943680a8ebdd7bd628190a2835ededc2"
meta-browser         = "HEAD:75640e14e325479c076b6272b646be7a239c18aa"
meta-gnome
meta-networking
meta-python
meta-filesystems     = "HEAD:8760facba1bceb299b3613b8955621ddaa3d4c3f"
meta-qt5             = "HEAD:d4e7f73d04e8448d326b6f89908701e304e37d65"‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

For info I have also tested on a clean build running the fsl-image-validation-imx image and get the same results as detailed above.

It was built using the imx-4.19.35-1.0.0.xml manifest from the imx-linux-thud branch

The BSP is: thud-4.19.35-1.0.0

Re: Gstreamer dropping frames (opencv videocapture)

Bio_TICFSL — Fri, 20 Mar 2020 18:00:00 GMT

Hi,

I have been trying to see this issues at OpenCV application, but without success.

However, I did the tests using GStreamer directly, and I am not seeing any dropped frame:

Waylandsink:

root@imx8mmevk:~# gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=120 ! "video/x-raw, width=320, height=240, framerate=(fraction)30/1" ! fpsdisplaysink sync=false video-sink="waylandsink"
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstWaylandSink:waylandsink0: sync = false
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, width=(int)320, height=(int)240
, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt7
09, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)320, height=(i
nt)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(stri
ng)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = vi
deo/x-raw, width=(int)320, height=(int)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ra
tio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:sr
c: caps = video/x-raw, width=(int)320, height=(int)240, format=(string)YUY2, framerate=(fraction)30/1, pix
el-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstWaylandSink:waylandsink0.GstPad:sink: caps = v
ideo/x-raw, width=(int)320, height=(int)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-r
atio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:vi
deo_sink: caps = video/x-raw, width=(int)320, height=(int)240, format=(string)YUY2, framerate=(fraction)30
/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)
320, height=(int)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, col
orimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)320, height=(
int)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(str
ing)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstWaylandSink:waylandsink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 17, dropped: 0, current: 31.74, average: 31.74
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 17, dropped: 0, current
: 31.74, average: 31.74
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 32, dropped: 0, current: 29.98, average: 30.89
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current
: 29.98, average: 30.89
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 47, dropped: 0, current: 30.00, average: 30.60
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 47, dropped: 0, current
: 30.00, average: 30.60
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 63, dropped: 0, current: 30.11, average: 30.47
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 63, dropped: 0, current
: 30.11, average: 30.47
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 78, dropped: 0, current: 30.00, average: 30.38
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 78, dropped: 0, current
: 30.00, average: 30.38
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 94, dropped: 0, current: 30.03, average: 30.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 94, dropped: 0, current
: 30.03, average: 30.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 109, dropped: 0, current: 30.00, average: 30.28
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 109, dropped: 0, curren
t: 30.00, average: 30.28
Got EOS from element "pipeline0".
Execution ended after 0:00:04.251347925
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (121), playing for (0:00:04.251560180), fps (28.460).
Freeing pipeline ...

Appsink:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...sync=false video-sink="wayland
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAppSink:appsink0: sync = false
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, width=(int)320, height=(int)240
, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt7
09, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)320, height=(i
nt)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(stri
ng)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = vi
deo/x-raw, width=(int)320, height=(int)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ra
tio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:sr
c: caps = video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComposition), width=(int)320, height=(int)2
40, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)b
t709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAppSink:appsink0.GstPad:sink: caps = video/x-r
aw(memory:SystemMemory, meta:GstVideoOverlayComposition), width=(int)320, height=(int)240, format=(string)
YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mod
e=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:sr
c: caps = video/x-raw, width=(int)320, height=(int)240, format=(string)YUY2, framerate=(fraction)30/1, pix
el-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAppSink:appsink0.GstPad:sink: caps = video/x-r
aw, width=(int)320, height=(int)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fr
action)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:vi
deo_sink: caps = video/x-raw, width=(int)320, height=(int)240, format=(string)YUY2, framerate=(fraction)30
/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)
320, height=(int)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, col
orimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)320, height=(
int)240, format=(string)YUY2, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(str
ing)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAppSink:appsink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 16, dropped: 0, current: 31.99, average: 31.99
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 16, dropped: 0, current
: 31.99, average: 31.99
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 31, dropped: 0, current: 29.98, average: 30.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 31, dropped: 0, current
: 29.98, average: 30.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 46, dropped: 0, current: 29.99, average: 30.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 46, dropped: 0, current
: 29.99, average: 30.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 61, dropped: 0, current: 30.00, average: 30.49
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 61, dropped: 0, current
: 30.00, average: 30.49
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 77, dropped: 0, current: 30.07, average: 30.40
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 77, dropped: 0, current
: 30.07, average: 30.40
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 93, dropped: 0, current: 30.07, average: 30.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 93, dropped: 0, current
: 30.07, average: 30.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = r
endered: 108, dropped: 0, current: 30.00, average: 30.30
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 108, dropped: 0, curren
t: 30.00, average: 30.30
^Chandling interrupt. (IT DOES NOT STOP ALONE!)
Interrupt: Stopping pipeline ...
Execution ended after 0:00:11.726109829
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

We do not have the same results than waylandsink, but note that every fdpdisplaysink print contains: dropped: 0.

So please, can you share the OpenCV application?

Plus, what is the camera used here? Note that 320x240 is a very low resolution, and it should not required the sync=false usage.

Regards

Re: Gstreamer dropping frames (opencv videocapture)

tyler_ward — Wed, 25 Mar 2020 15:17:19 GMT

Hi.

We are using the devkit camera module (MINISASTOCSI Product Information|NXP) which uses an OV5640 module The camera information is shown below if that is usefull, does this match with yours?

root@imx8mmevk:~# v4l2-ctl -D --list-formats-ext
Driver Info:
        Driver name      : mx6s-csi
        Card type        : i.MX6S_CSI
        Bus info         : platform:32e20000.csi1_bridge
        Driver version   : 4.19.35
        Capabilities     : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'YUYV' (YUYV 4:2:2)
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 320x240
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 720x480
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 2592x1944
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 0x0‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I have tested without the sync=false flag in the fpsdisplaysync and this makes no diference to the output.

Sharing the full CV application will be tricky (will need to sort out an NDA and make some changes so it can be compiled outside of our build enviroment) and most likley not perticualy usefull as it doesent give much in the way of diagnostic data. Our tests where we have tested the camera framerate (seperate from the processing logic) on the imx development board are carried out with the fps test program included in the previous post or the original performance test program which is very similar the one previous post but with some extra cpu performance tests added at the end (the fps test tool shared above is derived from the larger performance test program with the cpu tests deleted and the external functions contents moved inline). A new feature we added to the fps test tool since the original post is a per frame capture time, the test program posted before has now been expanded to include the delay in getting each frame to the output.

#include <cstdio>
#include <iostream>
#include <chrono>
#include <numeric>
#include <opencv2/core/mat.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/videoio.hpp>



int main(int argc, char* argv[])
{

    cv::VideoCapture cap(0);
    std::cout << cap.getBackendName() << std::endl;
    cap.set(cv::CAP_PROP_BUFFERSIZE, 2);
    cap.set(cv::CAP_PROP_FRAME_WIDTH, static_cast<double>(640) );
    cap.set(cv::CAP_PROP_FRAME_HEIGHT, static_cast<double>(480) );

    cv::Mat img;
    //perform a few initial captures to ensure correct setup
    for (int i = 0; i < 5; i++)
    {
        cap.read(img);
    }

    std::cout << "Camera framerate" << std::endl;

    //create time variables
    std::chrono::time_point<std::chrono::steady_clock> starttime;
    std::chrono::time_point<std::chrono::steady_clock> stoptime;

    starttime = std::chrono::steady_clock::now();
    for (int i = 0; i < 1000; i++)
    {
        std::chrono::time_point<std::chrono::steady_clock> framestarttime = std::chrono::steady_clock::now();
        cap.read(img);
        std::chrono::time_point<std::chrono::steady_clock> framestoptime = std::chrono::steady_clock::now();
        double time = ((double)std::chrono::duration_cast<std::chrono::milliseconds>(framestoptime - framestarttime).count()) / 1000;
        std::cout << time << std::endl;
    }
    stoptime = std::chrono::steady_clock::now();


    double time = ((double)std::chrono::duration_cast<std::chrono::milliseconds>(stoptime - starttime).count()) / 1000;
    double fps = 1000 / time;
    std::cout << "FPS = " << fps << "(" << time/200 << "s)" << std::endl;

}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The cut down output from the program above running with GST_DEBUG=2 is shown below.

0:00:00.054341855  8739     0x2892c980 WARN                    v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video0'.
0:00:00.911147891  8739     0x2893e800 WARN          v4l2bufferpool gstv4l2bufferpool.c:1223:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2_buffer.field to ANY
GSTREAMER
0:00:02.511571329  8739     0x2892c980 WARN                    v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video0'.
0:00:03.445758476  8739     0x2893e800 WARN          v4l2bufferpool gstv4l2bufferpool.c:1223:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2_buffer.field to ANY
0:00:05.041278859  8739     0x2892c980 WARN                    v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video0'.
0:00:05.918136869  8739     0x2893e800 WARN          v4l2bufferpool gstv4l2bufferpool.c:1223:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2_buffer.field to ANY
0:00:05.940245490  8739     0x2892c980 WARN                    v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video0'.
0:00:06.744731819  8739     0x2893e800 WARN          v4l2bufferpool gstv4l2bufferpool.c:1223:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2_buffer.field to ANY
0:00:06.854458540  8739     0x2893e800 WARN                 v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.913512905
0:00:06.964166886  8739     0x2893e800 WARN                 v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.023223905
Camera framerate
0.036
0:00:07.073884982  8739     0x2893e800 WARN                 v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.132903405
0.072
0.037
0:00:07.183563578  8739     0x2893e800 WARN                 v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.242613655
0.072
0.036
0:00:07.293251548  8739     0x2893e800 WARN                 v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.352293405
0.073
0.037

The above repeats several hundred times

0:01:01.484238939  8739     0x2893e800 WARN                 v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:55.543283280
0.072
0.037
0:01:01.593947160  8739     0x2893e800 WARN                 v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:55.652993280
0.071
0.037
0:01:01.703659756  8739     0x2893e800 WARN                 v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:55.762703405
0.072
0.037
0:01:01.813340727  8739     0x2893e800 WARN                 v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:55.872383530
0.072
FPS = 18.2319(0.274245s)





‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

If it is usefull I can send the Gstreamer debug log with a higher log level from running the application, just let me know where to send it.

If your not encountering the same issue could this be a defect in the development boards camera, is there a camera self test tool which can rule this in or out?

Re: Gstreamer dropping frames (opencv videocapture)

Bio_TICFSL — Wed, 22 Apr 2020 13:34:30 GMT

I did all the tests once more and I confirm there is no dropped frame from our side:

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstWaylandSink:waylandsink0: sync = false
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(se
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimee
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-e
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:src: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)e
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstWaylandSink:waylandsink0.GstPad:sink: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixele
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:video_sink: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(e
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-ratio=(fractione
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorime
0:00:00.431138500 4126 0xd2c7e30 WARN waylandsink wlwindow.c:333:gst_wl_window_new_internal: init surface_state fail, fallback to scale=1 fullscreen (1920x1048)
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstWaylandSink:waylandsink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 17, dropped: 0, current: 31.99, average: 31.99
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 17, dropped: 0, current: 31.99, average: 31.99
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 32, dropped: 0, current: 29.97, average: 31.01
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 29.97, average: 31.01
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 48, dropped: 0, current: 29.84, average: 30.61
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 48, dropped: 0, current: 29.84, average: 30.61
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 63, dropped: 0, current: 30.00, average: 30.46
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 63, dropped: 0, current: 30.00, average: 30.46
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 79, dropped: 0, current: 30.07, average: 30.38
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 79, dropped: 0, current: 30.07, average: 30.38
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 95, dropped: 0, current: 30.09, average: 30.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 95, dropped: 0, current: 30.09, average: 30.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 111, dropped: 0, current: 29.84, average: 30.26
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 111, dropped: 0, current: 29.84, average: 30.26
Got EOS from element "pipeline0".
Execution ended after 0:00:04.249358375
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (121), playing for (0:00:04.249576375), fps (28.473).
Freeing pipeline ...

root@imx8mmevk:~# gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=120 ! "video/x-raw, width=320, height=240, framerate=(fraction)30/1" ! fpsdisplaysink sync=false video-sink="appsink"
sdisplaysink sync=false video-sink="appsink"rate=(fraction)30/1" ! fps
Setting pipeline to PAUSED ...
0:00:00.148373875 4132 0x3fdf150 WARN v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev.
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAppSink:appsink0: sync = false
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(se
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimee
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-e
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:src: caps = video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComposition), width=(int)320, heige
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAppSink:appsink0.GstPad:sink: caps = video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComposition), width=(int)320, height=(int)240, framere
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:src: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)e
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAppSink:appsink0.GstPad:sink: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-e
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:video_sink: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(e
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-ratio=(fractione
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorime
0:00:00.159548125 4132 0x3f6ed90 WARN v4l2bufferpool gstv4l2bufferpool.c:811:gst_v4l2_buffer_pool_start:<v4l2src0:pool:src> Uncertain or not enough buffers, enabling copy threshold
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAppSink:appsink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 16, dropped: 0, current: 31.99, average: 31.99
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 16, dropped: 0, current: 31.99, average: 31.99
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 31, dropped: 0, current: 29.98, average: 30.99
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 31, dropped: 0, current: 29.98, average: 30.99
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 46, dropped: 0, current: 29.91, average: 30.63
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 46, dropped: 0, current: 29.91, average: 30.63
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 62, dropped: 0, current: 30.14, average: 30.50
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 62, dropped: 0, current: 30.14, average: 30.50
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 78, dropped: 0, current: 30.07, average: 30.41
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 78, dropped: 0, current: 30.07, average: 30.41
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 94, dropped: 0, current: 29.86, average: 30.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 94, dropped: 0, current: 29.86, average: 30.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 110, dropped: 0, current: 30.07, average: 30.28
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 110, dropped: 0, current: 30.07, average: 30.28
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:06.059819750
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

I was checking internally and we do not support the OpenCV application, but apparently there is no problem on it as well.

Maybe you could try to change the OpenCV.VideoCapture from v4l2 directly to a GStreamer pipeline and add some queues on it to avoid this kind of message, but as I said, there is no guarantee it will fix the issue, once we are not seeing it internally.

Please, let me know if it helps.

Re: Gstreamer dropping frames (opencv videocapture)

albanxiii — Wed, 29 Apr 2020 06:18:10 GMT

Hi BioTICFSL,

Hi all,

Could you please try tu run again this las command with GST_DEBUG set to 3, e.g.

GST_DEBUG=3 gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=120 ! "video/x-raw, width=320, height=240, framerate=(fraction)30/1" ! fpsdisplaysink sync=false video-sink="appsink"
sdisplaysink sync=false video-sink="appsink"rate=(fraction)30/1" ! fps

On my imx8mmevk + ov5640 board (MINISASTOCSI) I clearly see lost frames, but fpsdisplaysink does not seem to be aware of this.

An exerpt of the log :

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAppSink:appsink0: sync = false
0:00:01.155784521 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.064818513
0:00:01.265466476 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.174497138
0:00:01.375178307 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.284208138
0:00:01.484862263 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.393887513
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 11, dropped: 0, current: 21.45, average: 21.45
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 11, dropped: 0, current: 21.45, average: 21.45
0:00:01.594345091 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.503597638
0:00:01.704278924 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.613309262
0:00:01.813961380 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.722985513
0:00:01.923640960 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.832696263
0:00:02.032730409 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.941794388
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 21, dropped: 0, current: 18.25, average: 19.80
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 21, dropped: 0, current: 18.25, average: 19.80
0:00:02.142941620 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.052093763
0:00:02.252737702 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.161793513
0:00:02.362439533 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.271473263
0:00:02.472144864 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.381184388
0:00:02.581866695 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.490893513
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 31, dropped: 0, current: 18.20, average: 19.25
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 31, dropped: 0, current: 18.20, average: 19.25
0:00:02.691412274 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.600586638
0:00:02.801213106 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.710284388
0:00:02.911020313 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.819994638
0:00:03.020616017 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.929675513
0:00:03.130324473 4666 0x363e04f0 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.039381763

Using GST_DEBUG=4 will show you that v4l2src capture only get 2/3 of total frames. On can see 2 frames OK, 3rd frame lost, 2 frames ok, 3rd frame lost, and so on. This is surely a driver problem, because it is too much steasy rate.

I tried many pipeline configurations and observed that

gst source = v4l2src +gst sink = fakesink does not lose frames

gst source = videotestsrc +gst sink = appsink does not lose frames

gst source = v4l2src + gst sink = appsink does lose frames !!!

Obviously there is a problem with the combination of v4l2sr and appsink when both are used in gstreamer pipeline.

Re: Gstreamer dropping frames (opencv videocapture)

albanxiii — Wed, 29 Apr 2020 15:26:06 GMT

I think I an add kernel driver log showing that frames are missing at the driver level (mx6s_capture.c is compiled with DEBUG flag and debug is set to 2 in mxc_mipi_csi.c ) :

[86463.357435 < 0.023764>] mx6s-csi 32e20000.csi1_bridge: csi v4l2 busfreq high request.
[86463.358585 < 0.001150>] mx6s-csi 32e20000.csi1_bridge: No more fmt
[86463.358598 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: VIDIOC_CROPCAP not implemented
[86463.360416 < 0.001818>] mx6s-csi 32e20000.csi1_bridge: set to pixelformat 'YUYV-1'
[86464.114007 < 0.753591>] mx6s-csi 32e20000.csi1_bridge: count=4, size=0
[86464.114016 < 0.000009>] size=153600
[86464.119463 < 0.005447>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000c16dd1b5) 0x00000000ef9986ff 0
[86464.119510 < 0.000047>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000ad9f461d) 0x00000000494d1555 0
[86464.119540 < 0.000030>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86464.119569 < 0.000029>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86464.119615 < 0.000046>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000c16dd1b5) 0x00000000ef9986ff 153600
[86464.119621 < 0.000006>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000ad9f461d) 0x00000000494d1555 153600
[86464.119629 < 0.000008>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.119635 < 0.000006>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.132024 < 0.012389>] mxc_mipi-csi.0: mipi_csis_s_stream: 1, state: 0x1
[86464.132047 < 0.000023>] mxc_mipi-csi.0: fmt: 0x2008, 320 x 240
[86464.132061 < 0.000014>] ov5640_mipi 2-003c: s_stream: 1
[86464.241132 < 0.109071>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000c16dd1b5) 0x00000000ef9986ff 153600
[86464.277937 < 0.036805>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000ad9f461d) 0x00000000494d1555 153600
[86464.314490 < 0.036553>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.315481 < 0.000991>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86464.315494 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.351050 < 0.035556>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.352113 < 0.001063>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86464.352127 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.424197 < 0.072070>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.425150 < 0.000953>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86464.425163 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.460759 < 0.035596>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.461744 < 0.000985>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86464.461757 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.533876 < 0.072119>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.534830 < 0.000954>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86464.534845 < 0.000015>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.570435 < 0.035590>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.571388 < 0.000953>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86464.571402 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.643585 < 0.072183>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.644546 < 0.000961>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86464.644559 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.680145 < 0.035586>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.681092 < 0.000947>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86464.681105 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.753262 < 0.072157>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.754225 < 0.000963>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86464.754239 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.789859 < 0.035620>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.790819 < 0.000960>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86464.790832 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.862970 < 0.072138>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.863732 < 0.000762>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86464.863742 < 0.000010>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.899535 < 0.035793>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.900469 < 0.000934>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86464.900483 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86464.972681 < 0.072198>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86464.973642 < 0.000961>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86464.973656 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.009241 < 0.035585>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.010161 < 0.000920>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.010174 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.082357 < 0.072183>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.083325 < 0.000968>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86465.083338 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.118951 < 0.035613>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.119904 < 0.000953>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.119918 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.192066 < 0.072148>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.192999 < 0.000933>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86465.193013 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.228627 < 0.035614>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.229569 < 0.000942>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.229581 < 0.000012>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.301163 < 0.071582>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.302088 < 0.000925>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86465.302102 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.338334 < 0.036232>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.339308 < 0.000974>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.339321 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.411460 < 0.072139>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.412304 < 0.000844>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86465.412316 < 0.000012>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.448012 < 0.035696>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.448929 < 0.000917>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.448943 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.521159 < 0.072216>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.522091 < 0.000932>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86465.522104 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.557722 < 0.035618>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.558673 < 0.000951>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.558686 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.630838 < 0.072152>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.631791 < 0.000953>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86465.631805 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.667430 < 0.035625>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.668382 < 0.000952>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.668396 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.740547 < 0.072151>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.741495 < 0.000948>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86465.741509 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.777110 < 0.035601>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.778062 < 0.000952>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.778076 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.850255 < 0.072179>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.851219 < 0.000964>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86465.851232 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.886822 < 0.035590>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.887772 < 0.000950>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.887786 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.959945 < 0.072159>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.960769 < 0.000824>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86465.960781 < 0.000012>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86465.996527 < 0.035746>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86465.997459 < 0.000932>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86465.997472 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.069643 < 0.072171>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.070559 < 0.000916>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86466.070573 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.106204 < 0.035631>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.107165 < 0.000961>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86466.107179 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.179352 < 0.072173>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.180362 < 0.001010>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86466.180377 < 0.000015>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.215913 < 0.035536>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.216873 < 0.000960>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86466.216887 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.289031 < 0.072144>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.289958 < 0.000927>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86466.289973 < 0.000015>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.325595 < 0.035622>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.326538 < 0.000943>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86466.326552 < 0.000014>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.398736 < 0.072184>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.399664 < 0.000928>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86466.399677 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.435304 < 0.035627>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.436271 < 0.000967>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86466.436284 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.508424 < 0.072140>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.509249 < 0.000825>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000963a3046) 0x000000006f2e31f8 0
[86466.509261 < 0.000012>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000963a3046) 0x000000006f2e31f8 153600
[86466.545006 < 0.035745>] mx6s-csi 32e20000.csi1_bridge: mx6s_csi_frame_done (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.545953 < 0.000947>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000912735a2) 0x00000000dadf1f77 0
[86466.545966 < 0.000013>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000912735a2) 0x00000000dadf1f77 153600
[86466.585498 < 0.039532>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000c16dd1b5) 0x00000000ef9986ff 0
[86466.585514 < 0.000016>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000c16dd1b5) 0x00000000ef9986ff 153600
[86466.585534 < 0.000020>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_prepare (vb=0x00000000ad9f461d) 0x00000000494d1555 0
[86466.585541 < 0.000007>] mx6s-csi 32e20000.csi1_bridge: mx6s_videobuf_queue (vb=0x00000000ad9f461d) 0x00000000494d1555 153600
[86466.585581 < 0.000040>] mxc_mipi-csi.0: mipi_csis_s_stream: 0, state: 0x3
[86466.585589 < 0.000008>] ov5640_mipi 2-003c: s_stream: 0
[86466.591050 < 0.005461>] mxc_mipi-csi.0: Frame End events: 0
[86466.595585 < 0.004535>] mxc_mipi-csi.0: Frame Start events: 0
[86466.600289 < 0.004704>] mxc_mipi-csi.0: Non-image data after odd frame events: 0
[86466.606641 < 0.006352>] mxc_mipi-csi.0: Non-image data before odd frame events: 0
[86466.613078 < 0.006437>] mxc_mipi-csi.0: Non-image data after even frame events: 0
[86466.619518 < 0.006440>] mxc_mipi-csi.0: Non-image data before even frame events: 0
[86466.626043 < 0.006525>] mxc_mipi-csi.0: Unknown Error events: 0
[86466.630920 < 0.004877>] mxc_mipi-csi.0: CRC Error events: 0
[86466.635447 < 0.004527>] mxc_mipi-csi.0: ECC Error events: 0
[86466.639977 < 0.004530>] mxc_mipi-csi.0: FIFO Overflow Error events: 0
[86466.645375 < 0.005398>] mxc_mipi-csi.0: Lost Frame End Error events: 0
[86466.650858 < 0.005483>] mxc_mipi-csi.0: Lost Frame Start Error events: 0
[86466.656514 < 0.005656>] mxc_mipi-csi.0: SOT Error events: 0
[86466.662753 < 0.006239>] mx6s-csi 32e20000.csi1_bridge: csi v4l2 busfreq high release.

Just look at the time interval between successive mx6s_csi_frame_done and you'll see the problem.

At this point I just pointed a part of the problem, still have no solution.

Re: Gstreamer dropping frames (opencv videocapture)

tyler_ward — Fri, 01 May 2020 14:50:05 GMT

Can you let me know which operating system build you are using? If it was built by yocto if you can let me know the repo branch and manifest i can build that and test with the same version if not is there a download for the version you are using?

Re: Gstreamer dropping frames (opencv videocapture)

albanxiii — Wed, 06 May 2020 11:42:27 GMT

Hi Tyler,

My build system is Ubuntu 19.04. I build yocto image based on NXP i.Mx release 4.19.35_1.1.0

I copy/paste the quick start guide for your convenience :

Quick Start Guide
-----------------
See the i.MX Yocto Project User's Guide for instructions on installing repo.
First install the i.MX Linux BSP repo
$: repo init -u git://source.codeaurora.org/external/imx/imx-manifest.git -b imx-linux-warrior -m imx-4.19.35-1.1.0.xml
Download the Yocto Project Layers:
$: repo sync
If errors on repo init, remove the .repo directory and try repo init again.
Run i.MX Linux Yocto Project Setup:
$: [MACHINE=<machine>] [DISTRO=fsl-imx-<backend>] source ./fsl-setup-release.sh -b <build folder>
where
<machine> defaults to imx6qsabresd
<build folder> specifies the build folder name
After this your system will be configured to start a Yocto Project build.

Specifically image from which the log come from is buld according to this recipe (only relevant parts are showed here) :

inherit extrausers
require recipes-fsl/images/fsl-image-multimedia-full.bb
CORE_IMAGE_EXTRA_INSTALL += " \
v4l-utils \
opencv-samples \
systemd \
systemd-conf \
openssh \
v4l2loopback \
imx-gst1.0-plugin \
gstreamer1.0-libav \
gstreamer1.0-libav-dev \
gstreamer1.0-dev \
libopencv-core-dev \
libopencv-highgui-dev \
libopencv-imgproc-dev \
libopencv-objdetect-dev \
libopencv-ml-dev \
opencv-apps \
opencv-dev \
python-modules \
"

Additionnally as said it patched linux-imx (with a .bbappend) to enable low level CSI/MIPI debug :

diff --git a/drivers/media/platform/mxc/capture/mx6s_capture.c b/drivers/media/platform/mxc/capture/mx6s_capture.c
index f57817ab2647..d65d0f15d62d 100644
--- a/drivers/media/platform/mxc/capture/mx6s_capture.c
+++ b/drivers/media/platform/mxc/capture/mx6s_capture.c
@@ -18,6 +18,9 @@
*
* @ingroup CSI
*/
+
+#define DEBUG
+
#include <asm/dma.h>
#include <linux/busfreq-imx.h>
#include <linux/clk.h>
@@ -756,6 +759,8 @@ static int mx6s_csi_enable(struct mx6s_csi_dev *csi_dev)
if (pix->field == V4L2_FIELD_INTERLACED)
csi_tvdec_enable(csi_dev, true);
+ else
+ csi_tvdec_enable(csi_dev, false);
/* For mipi csi input only */
if (csi_dev->csi_mipi_mode == true) {
diff --git a/drivers/media/platform/mxc/capture/mxc_mipi_csi.c b/drivers/media/platform/mxc/capture/mxc_mipi_csi.c
index 0e133301a49f..154fb0c004c6 100644
--- a/drivers/media/platform/mxc/capture/mxc_mipi_csi.c
+++ b/drivers/media/platform/mxc/capture/mxc_mipi_csi.c
@@ -46,7 +46,7 @@
#include <media/v4l2-subdev.h>
#include <media/v4l2-device.h>
-static int debug;
+static int debug = 2;
module_param(debug, int, 0644);
MODULE_PARM_DESC(debug, "Debug level (0-2)");

At th end I wonder if anyone uses iMX8M mini as a camera / video capture system...

Hope it helps.

Re: Gstreamer dropping frames (opencv videocapture)

albanxiii — Tue, 09 Jun 2020 06:25:54 GMT

Update : look at https://community.nxp.com/message/1018843?commentID=1018843#comment-1018843

This solves my problem

https://community.nxp.com/message/1018843?commentID=1018843#comment-1018718

- Please enlarge GST_V4L2_MIN_BUFFERS in the following file from 2 to 7
/gstreamer1.0-plugins-good/1.12.2-r0/git/sys/v4l2/gstv4l2object.h :
#define GST_V4L2_MIN_BUFFERS 7

https://community.nxp.com/message/1018843?commentID=1018843#comment-1018843

And please enable dma-coherent in fsl-imx8mq.dtsi for csix_bridge:

csi1_bridge: csi1_bridge@30a90000 {
compatible = "fsl,imx8mq-csi", "fsl,imx6s-csi";
reg = <0x0 0x30a90000 0x0 0x10000>;
interrupts = <GIC_SPI 42 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&clk IMX8MQ_CLK_DUMMY>,
<&clk IMX8MQ_CLK_CSI1_ROOT>,
<&clk IMX8MQ_CLK_DUMMY>;
clock-names = "disp-axi", "csi_mclk", "disp_dcic";
dma-coherent;
status = "disabled";
};

Hope it helps.

Re: Gstreamer dropping frames (opencv videocapture)

tyler_ward — Thu, 11 Jun 2020 14:36:51 GMT

Thanks for finding that.

Adding those patches fixed the issue (tested on the 4.19.35 bsp) with the gstreamer commandline tests but caused issues with getting the data in opencv if the buffer was too long (log from buffer length 5 coppied below if usefull to anyone). Setting the gstreamer buffer from 2 to either 3 or 4 worked for both gstreamer on the command line and opencv though.

0:00:00.349274292 4901 0x21e3bc60 WARN v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video0'.
0:00:01.017272128 4901 0x21f16800 WARN v4l2bufferpool gstv4l2bufferpool.c:1223:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2_buffer.field to ANY
GSTREAMER
Message from syslogd@imx8mmevk at Thu Jun 11 14:29:49 2020 ...
imx8mmevk kernel: [ 1548.003012] Internal error: Oops: 96000006 [#1] PREEMPT SMP
Message from syslogd@imx8mmevk at Thu Jun 11 14:29:49 2020 ...
imx8mmevk kernel: [ 1548.208794] Code: 35ffff91 aa1003e0 d65f03c0 f9800011 (885ffc10)
Message from syslogd@imx8mmevk at Thu Jun 11 14:29:49 2020 ...
imx8mmevk kernel: [ 1548.017115] Process visionrunner_pe (pid: 4901, stack limit = 0x00000000ad8b3b9f)