i.MX8QXP: unable to link v4l2 decoder to encoder in the same gstreamer pipeline

[gabrielvalcazar]

Good afternoon,

I'm currently testing the multimedia capabilities of an i.MX8QXP-based system running a custom Yocto image based on the 5.4.47_2.2.0​​ BSP. In one of my tests, I tried queuing the output of the v4l2 JPEG decoder (I assume it's the element I want to use if I'm interested in testing the 8X's VPU) into the v4l2 encoder. Here is a simplified example of what I want to test:

 gst-launch-1.0 v4l2src device=/dev/video3 ! v4l2jpegdec ! v4l2h264enc ! fakesink

 The command fails with the following error:

  ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
   ../../../../git/libs/gst/base/gstbasesrc.c(3072): gst_base_src_loop ():    /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
   streaming stopped, reason not-negotiated (-4)
   Execution ended after 0:00:01.074351570
   Setting pipeline to PAUSED ...
   Setting pipeline to READY ...
   Setting pipeline to NULL ...
   Freeing pipeline ...

In another example, I'm able to use the v4l2 video decoder and encoder separately, but not together. These two commands work separately:

gst-launch-1.0 -e videotestsrc ! v4l2h264enc ! h264parse ! mp4mux ! filesink location=test.mp4

gst-launch-1.0 -v filesrc location=test.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! testsink

 But trying to use both elements in one pipeline fails:

gst-launch-1.0 -v filesrc location=test.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! v4l2h264enc ! testsink

What is the reason for this? Is there a missing element in the pipeline, or is this a known limitation (eiher for gstreamer or for the i.MX8QXP's VPU)?

Thanks in advance,

Gabriel

[banhu]

Hi , 

The same issue on iMX8QM with 5.4.70_2.3.1. It is fine to preview a USB camera.

gst-launch-1.0 v4l2src device=/dev/video2 ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! v4l2jpegdec ! waylandsink sync=false

But it just fails when encoding v4l2jpegdec's src stream into H.264 by v4l2jpegdec. The v4l2jpegdec's src format(video/x-raw, NV12)  meets v4l2jpegdec sink format requirement. 

gst-launch-1.0 v4l2src device=/dev/video2 ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! v4l2jpegdec ! video/x-raw, format=NV12, width=1280, height=720, framerate=30/1 ! v4l2h264enc ! fakesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../git/libs/gst/base/gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.647766125
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

[Hongtao]

I have the same problem as you.

In addition, problems can occur when using both v4l2dec and imxcompositor.

V4l2src This route shows that stopping will be blocked

  gst-launch-1.0 imxcompositor_g2d name=comp sink_1::xpos=0 sink_1::ypos=0 !  waylandsink   v4l2src  device=/dev/video4  ! video/x-h264,width=1280,height=720,framerate=30/1 ! v4l2h264dec ! comp.sink_0   videotestsrc ! comp.sink_1
  
    gst-launch-1.0 imxcompositor_g2d name=comp sink_1::xpos=0 sink_1::ypos=0 !  waylandsink   v4l2src  device=/dev/video2 ! image/jpeg,width=1280,height=720,framerate=30/1 ! v4l2jpegdec ! comp.sink_0   videotestsrc ! comp.sink_1

It works by decoding with jpegdec software, but it is extremely unstable and shows Carton frequently

	   gst-launch-1.0 imxcompositor_g2d name=comp sink_1::xpos=0 sink_1::ypos=0 !  waylandsink   v4l2src  device=/dev/video2 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegdec ! comp.sink_0   videotestsrc ! comp.sink_1

[gabrielvalcazar]

Hi again,

I've returned to investigating this issue after some time. I'm currently using the v5.4.70_2.3.1 BSP and I'm still running into the same problem.

My final goal is to be able to transcode a JPEG camera feed into an H.264 format with the least possible amount of CPU overhead. As reference, I'm using the example in section 9.1 of the i.MX 8 GStreamer user guide, which is supposed to accomplish what I'm looking for:

gst-launch-1.0 v4l2src device=/dev/video3 ! jpegparse ! v4l2jpegdec ! queue ! videoconvert ! v4l2h264enc ! h264parse ! matroskamux ! filesink location=out.mkv

 However, this pipeline doesn't work on my i.MX8QXP device.

The following pipeline does seem to go through, but no video stream is stored in the out.mkv file and the gst-launch-1.0 app blocks when stopping the stream:

gst-launch-1.0 v4l2src device=/dev/video3 ! jpegparse ! v4l2jpegdec ! imxvideoconvert_g2d ! v4l2convert ! v4l2h264enc ! h264parse ! matroskamux ! filesink location=out.mkv
Setting pipeline to PAUSED ...
[   49.299976] uvcvideo: Failed to query (GET_DEF) UVC control 12 on unit 2: -32 (exp. 4).
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[   49.926396] [VPU Decoder] firmware version is 1.7.0
Redistribute latency...
[   49.960932] bypass csc
[   49.963307] input fmt YUYV
[   49.966092] output fmt NV12
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:03.492342625
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
^C^C^C^C^C

 I've also tried piping the stream to the waylandsink, but I'm unable to see anything on screen. I've also tried the suggestion of hardcoding a 1280x720 resolution in the pipeline, but that doesn't work.

Is there a pipeline that will allow me to transcode a JPEG feed to H.264 while avoiding CPU overhead (e.g. using the VPU when possible or zero-copy methods like the ones suggested in the GStreamer user guide)?

[toshinariagata]

Hi,

I have the same problem.
Has there been any progress?

[joanxie]

how about trying the command as below:

gst-launch-1.0 -v filesrc location=test.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! v4l2convert ! v4l2h264enc ! testsink

[gabrielvalcazar]

Hi joanxie,

Using the command you suggested, gst-launch-1.0 no longer fails with an error and I see kernel messages that suggest that both the decoder and the encoder are being used.

However, when I try stopping the command with Ctrl+C, an error appears and I'm unable to terminate the command properly, so I'm stuck until I reset the target. Here's the output:

~# gst-launch-1.0 -e filesrc location=test.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! v4l2convert ! v4l2h264enc ! h264parse ! mp4mux ! filesink location="result.mp4"
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
[  246.781463] [VPU Decoder] firmware version is 1.6.4
Redistribute latency...
[  246.807488] bypass csc
[  246.809924] input fmt YUYV
[  246.812685] output fmt NV12
Redistribute latency...

^Chandling interrupt.
Interrupt: Stopping pipeline ...
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...

 I feel like this pipeline is close to a proper solution, but it's still isn't working as expected.

[songhongliang]

HI,

gst-launch-1.0 imxcompositor_g2d v4l2src device=/dev/video8 ! 'video/x-raw,framerate=25/1,width=1280,height=720' ! autovideosink

I think the width=1280,height=720 is very important，You need to check the available supported resolutions of the video