i.MX8X: transcode m-jpeg video to h264 with gstreamer using the VPU

[gabrielvalcazar]

Greetings,

We are using a custom board/OS based on the i.MX8 QXP C0 and BSP v5.4.70_2.3.1. Our goal for the past several months has been to transcode an m-jpeg video (from a live camera stream, but we're testing with a pre-made file for now: https://digi.app.box.com/s/v66elvh2y4d9bbe9nud5og14o6ufdehg) to an h.264 MP4/Matroska container. The idea is to make use of the VPU both when decoding and encoding to remove as much CPU overhead as possible.

We've tried several different pipelines, but all of them fail when negotiating the caps between the video format converter and the h.264 encoder:

gst-launch-1.0 -ve filesrc location=OBC_mjpeg.avi ! avidemux ! v4l2video1jpegdec ! imxvideoconvert_g2d ! v4l2h264enc ! h264parse ! mp4mux ! filesink location=OBC_h264.mp4

gst-launch-1.0 -ve filesrc location=OBC_mjpeg.avi ! avidemux ! v4l2video1jpegdec ! videoconvert ! v4l2h264enc ! h264parse ! mp4mux ! filesink location=OBC_h264.mp4

gst-launch-1.0 -ve filesrc location=OBC_mjpeg.avi ! avidemux ! v4l2jpegdec ! imxvideoconvert_g2d ! v4l2h264enc ! h264parse ! mp4mux ! filesink location=OBC_h264.mp4

gst-launch-1.0 -ve filesrc location=OBC_mjpeg.avi ! avidemux ! v4l2jpegdec ! videoconvert ! v4l2h264enc ! h264parse ! mp4mux ! filesink location=OBC_h264.mp4

 We believe this is because one or more of the capabilities such as colorimetry or pixel-aspect-ratio are causing nothing to pass the caps selection filter.

Enforcing the caps in the pipeline doesn't work either:

gst-launch-1.0 -ve filesrc location=OBC_mjpeg.avi ! avidemux ! v4l2video1jpegdec ! imxvideoconvert_g2d ! video/x-raw, format=NV12, width=1920, height=1080 ! v4l2h264enc ! h264parse ! mp4mux ! filesink location=5_test.mp4

 See the pipeline graph, where the format is set to RGB in the first half and NV12 in the second half:

When playing the m-jpeg video using the JPEG decoder, you can see the video just fine and the pipeline uses the NV12 format:

gst-launch-1.0 -ve filesrc location=OBC_mjpeg.avi ! avidemux ! v4l2video1jpegdec ! imxvideoconvert_g2d ! videoscale ! waylandsink sync=false qos=false

Graph:

We've also tried using the zero-copy pipelines suggested in the i.MX gstreamer manual, but those don't work in our case.

Could you help us understand the reason of this issue? Is there a specific pipeline that can achieve what we're looking for (m-jpeg to h264 transcoding with VPU/JPEG acceleration)?

Best regards,

Gabriel

[joanxie]

it seems that imxvideoconvert_g2d couldn't output NV12, try to use imxvideoconvert_g2d ! v4l2convert to add ISI convert

[gabrielvalcazar]

Hi @joanxie,

Thank for your feedback. It seems like adding v4l2convert to the pipeline after imxvideoconvert_g2d changes the behavior, but now gstreamer freezes and I have no way of stopping it (not even with "kill") or launching another instance of gstreamer (I get errors about ISI being busy if I try). The only way to stop it is by resetting the target. Here's the output I get:

~# gst-launch-1.0 -ve filesrc location=OBC_mjpeg.avi ! avidemux ! v4l2video1jpegdec ! imxvideoconvert_g2d ! v4l2convert ! v4l2h264enc ! h264parse ! mp4mux ! filesink location=OBC_h264.mp4

Setting pipeline to PAUSED ...

Pipeline is PREROLLING ...

/GstPipeline:pipeline0/v4l2video1jpegdec:v4l2video1jpegdec0.GstPad:sink: caps = image/jpeg, framerate=(fraction)30/1, width=(int)1920, height=(int)1080

/GstPipeline:pipeline0/v4l2video1jpegdec:v4l2video1jpegdec0.GstPad:src: caps = video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1

/GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1, format=(string)YUY2

/GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:src: caps = video/x-raw, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, format=(string)NV12, width=(int)1920, height=(int)1080, colorimetry=(string)bt709

/GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)baseline, level=(string)4, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)baseline, level=(string)4, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

Redistribute latency...

/GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:sink: caps = video/x-raw, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, format=(string)NV12, width=(int)1920, height=(int)1080, colorimetry=(string)bt709

/GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1, format=(string)YUY2

/GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0.GstPad:sink: caps = video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1

 Here are the graphical representations of the pipelines I tried (one uses v4l2jpecdec, and the other uses v4l2video1jpegdec):

Any reason as to why gstreamer is freezing after adding v4l2convert?

Thanks in advance,

Gabriel

[gabrielvalcazar]

Hi again @joanxie ,

Is there any advice you could give regarding the behavior I explained in my post above? v4l2convert makes the pipeline start, but it freezes and there's no way to kill it afterwards.

Is m-jpeg to h264 transcoding possible with the i.MX8X VPU? If so, is there a way to do it with minimal CPU overhead?

Thanks and best regards,

Gabriel

[joanxie]

what commands do you use? let me reproduce this

[gabrielvalcazar]

@joanxie 

You can find the command in the code block of my previous reply:

gst-launch-1.0 -ve filesrc location=OBC_mjpeg.avi ! avidemux ! v4l2video1jpegdec ! imxvideoconvert_g2d ! v4l2convert ! v4l2h264enc ! h264parse ! mp4mux ! filesink location=OBC_h264.mp4

 Replace the .avi file with an .avi file of your choice

[gabrielvalcazar]

Hi @joanxie ,

Just so you know, I've tried the same pipeline using both v4l2video1jpegdec and v4l2jpegdec, with the same results.

Any advice moving forward?

Thanks in advance,

Gabriel

[joanxie]

I have tried replaced  v4l2video1jpegdec to v4l2jpegdec successfully, pls try it

[gabrielvalcazar]

Hi @joanxie ,

I believe there has been a misunderstanding. I've already tried using that decoder, and I still see the same behavior. Note that I'm still using the same setup as the one in my original post (i.MX8 QXP C0 and Linux BSP v5.4.70_2.3.1).

When you say that changing the decoder works for you: which BSP are you using? Which i.MX SOC? Which gstreamer pipeline?

My main goal is still the same: transcode an m-jpeg video to h264 with minimal CPU overhead.

Thanks in advance,

Gabriel

[joanxie]

firstly I tried your command as below failed, and the error message like

ERROR: from element /GstPipeline:pipeline0/GstMP4Mux:mp4mux0: Could not multiplex stream.
Additional debug info:
../../../git/gst/isomp4/gstqtmux.c(4832): gst_qt_mux_add_buffer (): /GstPipeline:pipeline0/GstMP4Mux:mp4mux0:
Buffer has no PTS.
EOS on shutdown enabled -- waiting for EOS after Error
Waiting for EOS...
ERROR: from element /GstPipeline:pipeline0/GstAviDemux:avidemux0: Internal data stream error.
Additional debug info:
../../../git/gst/avi/gstavidemux.c(5780): gst_avi_demux_loop (): /GstPipeline:pipeline0/GstAviDemux:avidemux0:
streaming stopped, reason error (-5)

then change the v4l2video1jpegdec , then no errors shows, my board is C0 too, I used the 5.10 bsp, but the 5.4 should be same,

gst-launch-1.0 -ve filesrc location=OBC_mjpeg.avi ! avidemux ! v4l2video1jpegdec ! imxvideoconvert_g2d ! v4l2convert ! v4l2h264enc ! h264parse ! mp4mux ! filesink location=OBC_h264.mp4

[gabrielvalcazar]

Hi @Joan ,

I managed to get my hands on a i.MX8QXP MEK board. Even though it has a B0 SOC revision, I installed the prebuilt NXP v5.4.70 embedded Linux images and I was able to get it to work on my first try - v4l2video1jpegdec doesn't work but v4l2jpegdec does, as you mention.

Since the VPU interface is internal to the SOC; I doubt the issue is due to the hardware on my board, but rather the software. After all, I'm using a modified version of NXP's v5.4.70 BSP.

I suspect the issue may lie in one or several of the following components:

    - gstreamer plugins

    - VPU driver/firmware (I doubt it, because we don't modify any of them)

    - VPU memory regions

    - Device tree configuration

    - JPEG decoder (I doubt it, because the behavior I see on my board is the same regardless of using the VPU or the JPEG decoder)

Now that it's obvious that the problem is strictly on my end, this thread can be closed, but before doing so - are there any more software components involved in this use case that I should be aware of?

Thank you very much for your help and your patience,

Gabriel

[gabrielvalcazar · Accepted solution]

Hi @joanxie ,

I found the issue with my board, it was in the device tree. All I had to do was enable the m2m_device
node in the isi_0 interface. Now the transcoding works perfectly.

I would like to iron out some performance issues I've seen, but I created a separate thread for them as to not clog this current thread with more replies: https://community.nxp.com/t5/i-MX-Processors/i-MX8X-m-jpeg-to-h264-VPU-transcoding-speed/td-p/129066...

Once again, thank you for your patience and your help.

Best regards,

Gabriel

[alexstefan]

Hi @gabrielvalcazar ,

How did you enable the m2m_device node? I think I'm having similar issues trying to decode and then re-encode to H264 in  the same pipeline.

Thanks!

[joanxie]

the picture is too small to see, do you mind uploading the normal size?

[gabrielvalcazar]

Hi,

You should be able to view the images in their original size by downloading them. Click an image, then click the "download image" button on the right.

Best regards,

Gabriel