Problem with v4l2src and vpuenc

[michaelb_]

Hi

I have a quad-core imx6 board. Using Yocto master-next I am experiencing a strange problem with my video cam. The videocam is a Logitech C270.

What is working:

gst-launch v4l2src device=/dev/video0 ! mfw_v4lsink

working nicely, getting highest resolution, low fps, 20-30% CPU usage (one core).

gst-launch v4l2src device=/dev/video0 ! video/x-raw-yuv, width=640,height=480,framerate=25/1 ! mfw_v4lsink

working even better, lower resolution, but nice framerate, again about 20-25% CPU usage.

Now, when I try to encode the video, then I get problems:

gst-launch v4l2src ! 'video/x-raw-yuv,format=(fourcc)I420,width=640,height=480,framerate=25/1' ! queue max-size-buffers=2 ! vpuenc codec=6 ! matroskamux ! filesink location=output.mkv sync=false

Setting pipeline to PAUSED ...

[INFO]    Product Info: i.MX6Q/D/S

vpuenc versions :smileyhappy:

    plugin: 3.0.10

    wrapper: 1.0.45(VPUWRAPPER_ARM_LINUX Build on Mar 22 2014 11:51:52)

    vpulib: 5.4.20

    firmware: 2.3.10.40778

ERROR: Pipeline doesn't want to pause.

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Could not negotiate format

Additional debug info:

/home/michael/matrix-yocto/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer/0.10.36-r2/gstreamer-0.10.36/libs/gst/base/gstbasesrc.c(2830): gst_base_src_start (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:

Check your filtered caps, if any

Setting pipeline to NULL ...

Freeing pipeline ...

I can only get that to work if I add a 'ffmpegcolorspace' element after the v4l2src. This, however, gives directly more than 100% CPU usage. It works, but very inefficiently.

Now the question: Am I doing something wrong? How can I get the encode working with normal CPU usage?

Thanks for any help.

Michael

[aurelienbouin]

exactly the same problem for me ... did you find what was wrong ?

[michaelb_]

Hmm, however, what I don't understand then is this:

gst-inspect mfw_v4lsink| egrep 'format|video/'

      video/x-raw-yuv

                 format: NV12

      video/x-raw-yuv

                 format: I420

      video/x-raw-yuv

                 format: TNVP

      video/x-raw-yuv

                 format: TNVF

      video/x-raw-yuv

                 format: YV12

      video/x-raw-yuv

                 format: Y42B

      video/x-raw-yuv

                 format: Y444

      video/x-raw-yuv

                 format: Y800

      video/x-raw-yuv

                 format: YV12

      video/x-raw-yuv

                 format: UYVY

      video/x-raw-yuv

                 format: YUY2

      video/x-raw-rgb

      video/x-raw-rgb

      video/x-raw-rgb

      video/x-raw-rgb

      video/x-raw-rgb

This means that

gst-launch v4l2src ! video/x-raw-yuv, width=640,height=480,framerate=25/1 ! mfw_v4lsink axis-left=100 axis-top=260 disp-width=940 disp-height=640

should not work either. But it does. Hmmm?

[LeonardoSandova]

?Michael, use mfw_ipucsc instead of ffmpeg element. This way you get HW acceleration.

Leo

[michaelb_]

Hi Leo

ipucsc looks like an interesting element. I can't get it working, though.

This pipeline works:

gst-launch v4l2src ! ffmpegcolorspace ! 'video/x-raw-yuv,format=(fourcc)I420,width=640,height=480,framerate=25/1' ! queue max-size-buffers=2 ! vpuenc codec=6 ! matroskamux ! filesink location=output.mkv sync=false

This does not work (same error as above: ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Could not negotiate format):

gst-launch v4l2src ! mfw_ipucsc ! 'video/x-raw-yuv,format=(fourcc)I420,width=640,height=480,framerate=25/1' ! queue max-size-buffers=2 ! vpuenc codec=6 ! matroskamux ! filesink location=output.mkv sync=false

There don't seem to be any parameters to ipucsc, so I don't know, what I do wrong. I did try moving the ipucsc to other places in the pipeline, but without any success.

What is wrong?

Btw. very strange behavior of the pipe when running an eglfs application in the background:

gst-launch v4l2src ! ffmpegcolorspace ! 'video/x-raw-yuv,format=(fourcc)I420,width=640,height=480,framerate=25/1' ! queue max-size-buffers=2 ! vpuenc codec=6 ! matroskamux ! filesink location=output.mkv sync=false

uses 8% of CPU when run alone. Together with a (simple) eglfs application (with a simple GL Shader running), the thread increases its own CPU usage to 15-20%.

Same applies to this:

gst-launch v4l2src ! ffmpegcolorspace ! 'video/x-raw-yuv,format=(fourcc)I420,width=640,height=480,framerate=25/1' ! queue max-size-buffers=2 ! vpuenc codec=6 ! matroskamux ! filesink location=output.mkv sync=false

uses 25-30%, when run alone. Together with the eglfs application, it increases its usage to 50-60% (or sometimes nearly to 100%).

[LeonardoSandova]

did you try with mfw_ipucsc element? unless you have create an alias, use this Freescale element.

?

[michaelb_]

Hi Leo

Sorry, it was very late yesterday, when I wrote the last post.

Yes, I used mfw_ipucsc, not ipucsc (the latter would have given an element not found error, I guess).

I have now corrected the post to see the pipes I used.

[LeonardoSandova]

the mfw_ipucsc element can can accept (sink caps) and provide (source caps) these type of frame's format

root@imx6qsabresd:~# gst-inspect  mfw_ipucsc | egrep 'format|video/'

  Description: Convert video raw data between different formats and resolutions by using IPU

      video/x-raw-yuv

                 format: NV12

      video/x-raw-yuv

                 format: I420

      video/x-raw-yuv

                 format: UYUV

      video/x-raw-rgb

      video/x-raw-yuv

                 format: NV12

      video/x-raw-yuv

                 format: I420

      video/x-raw-yuv

                 format: UYUV

      video/x-raw-rgb

Can you add '-v' to your pipeline and see what are the caps being negotiated?

[michaelb_]

Hi Leo

Debugging gstreamer does not seem to be as simple as thought. Adding "-v" (once or several times) does not alter the output:

gst-launch -v v4l2src ! mfw_ipucsc ! 'video/x-raw-yuv,format=(fourcc)I420,width=640,height=480,framerate=25/1' ! queue max-size-buffers=2 ! vpuenc codec=6 ! matroskamux ! filesink location=output.mkv sync=false

IPU_CSC_CORE_LIBRARY_VERSION_INFOR_01.00.

MFW_GST_IPU_CSC_PLUGIN 3.0.10 build on Apr 15 2014 10:00:31.

Setting pipeline to PAUSED ...

[INFO]    Product Info: i.MX6Q/D/S

vpuenc versions :smileyhappy:

    plugin: 3.0.10

    wrapper: 1.0.45(VPUWRAPPER_ARM_LINUX Build on Apr 15 2014 09:52:34)

    vpulib: 5.4.20

    firmware: 2.3.10.40778

ERROR: Pipeline doesn't want to pause.

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Could not negotiate format

Additional debug info:

/home/ubuntu/yocto/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer/0.10.36-r2/gstreamer-0.10.36/libs/gst/base/gstbasesrc.c(2830): gst_base_src_start (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:

Check your filtered caps, if any

Setting pipeline to NULL ...

Freeing pipeline ...

I went through the debugging tutorials and came up with something like this:

gst-launch v4l2src ! mfw_ipucsc ! 'video/x-raw-yuv,format=(fourcc)I420,width=640,height=480,framerate=25/1' ! queue max-size-buffers=2 ! vpuenc codec=6 ! matroskamux ! filesink location=output.mkv sync=false --gst-debug=v4l2:3

IPU_CSC_CORE_LIBRARY_VERSION_INFOR_01.00.

MFW_GST_IPU_CSC_PLUGIN 3.0.10 build on Apr 15 2014 10:00:31.

Setting pipeline to PAUSED ...

0:00:00.689357000  9989  0x1e3a8a0 INFO                    v4l2 /home/ubuntu/yocto/boundary-3.10.17/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gst-plugins-good/0.10.31-r8/gst-plugins-good-0.10.31/sys/v4l2/v4l2_calls.c:485:gst_v4l2_open:<v4l2src0> Opened device 'UVC Camera (046d:0825)' (/dev/video0) successfully

[INFO]    Product Info: i.MX6Q/D/S

vpuenc versions :smileyhappy:

    plugin: 3.0.10

    wrapper: 1.0.45(VPUWRAPPER_ARM_LINUX Build on Apr 15 2014 09:52:34)

    vpulib: 5.4.20

    firmware: 2.3.10.40778

0:00:00.708444333  9989  0x1e3a8a0 INFO                    v4l2 /home/ubuntu/yocto/boundary-3.10.17/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gst-plugins-good/0.10.31-r8/gst-plugins-good-0.10.31/sys/v4l2/gstv4l2object.c:1105:gst_v4l2_object_fill_format_list:<v4l2src0> got 2 format(s):

0:00:00.708704667  9989  0x1e3a8a0 INFO                    v4l2 /home/ubuntu/yocto/boundary-3.10.17/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gst-plugins-good/0.10.31-r8/gst-plugins-good-0.10.31/sys/v4l2/gstv4l2object.c:1111:gst_v4l2_object_fill_format_list:<v4l2src0>   YUYV

0:00:00.708885667  9989  0x1e3a8a0 INFO                    v4l2 /home/ubuntu/yocto/boundary-3.10.17/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gst-plugins-good/0.10.31-r8/gst-plugins-good-0.10.31/sys/v4l2/gstv4l2object.c:1111:gst_v4l2_object_fill_format_list:<v4l2src0>   MJPG

ERROR: Pipeline doesn't want to pause.

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Could not negotiate format

Additional debug info:

/home/ubuntu/yocto/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer/0.10.36-r2/gstreamer-0.10.36/libs/gst/base/gstbasesrc.c(2830): gst_base_src_start (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:

Check your filtered caps, if any

Setting pipeline to NULL ...

Freeing pipeline ...

If I use "--gst-debug=3", I get very long debug output.

Do I read this correctly that only two output formats are possible YUYV and MJPG, both of which are not supported by mfw_ipucsc?

If this is the problem, I guess I should get a different webcam then :-)

[LeonardoSandova]

I am not familiar with color space formats :smileysad: perhaps YUVV is the same as one compatible with the ipucsc but names are different.

[michaelb_]

I don't know anything about the color space formats either. I found one reference here (https://wiki.videolan.org/YUV/#YUY2) hinting that YUY2 and YUYV are the same. That would explain why mfw_v4lsink (which accepts YUY2) can accept the webcam (which outputs YUYV) directly, but vpuenc as well as mfw_ipucsc both need ffmpegcolorspace conversion.

Unfortunately the webcam producers don't write (even in their "technical docs" what color fomats their products support). So, I can't simply purchase a new one. It might just support the same formats.

Anyway, thanks a lot for your help!!!