imx8mp problems with vpudec and tee

[rradman_zenitel]

Hi,

I'm trying to create a GStreamer pipeline that decodes H264 video using vpudec and tee elements looking something like this:

v4l2src ! vpudec ! tee name=t

t. ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! h264parse ! rtph264pay ! rtpbin ! udpsink
t. ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! appsink

If I run the pipeline with this configuration while attaching only one of the branches to the tee elements, it works fine. As soon as I add the second branch to the tee element (ordering doesn't matter), my program crashes. I've tried adding queue elements with max-size-time=0 property to the first pipeline (before or after the vpudec, or both) but It did nothing.

I've managed to prevent the crashing by adding more elements to the first pipeline so it would look like this:

v4l2src ! vpudec ! queue ! videorate ! imxvideoconvert_g2d ! tee

By doing so, a new problem appears - Adding a second branch with lower video resolution affects the video resolution on the first branch.

Steps to reproduce:
- add branch 1 at 1080p

- add branch 2 at 240p

Result: Branch 1 resolution drops significantly. Looks like 240p.

This happens on-the-fly, without restarting the pipelines.

Note: When running the similar configuration with RAW camera on the same device (no decoding needed), there is no any issues.

v4l2src ! tee

EDIT: I've added logs as an attachment.

Pipeline used:

GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-h264 ! vpudec ! tee name=t 
t. ! queue ! videorate ! imxvideoconvert_g2d ! videoconvert ! vpuenc_h264 ! appsink 
t. ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! h264parse ! rtph264pay ! udpsink 

Individually both branches work fine, program crashes when adding second branch.

Every advice is much appreciated.

If needed I can provide more information.

Thanks!

Robert

[rradman_zenitel · Accepted solution]

Issue was fixed by installing the new decoder (v4l2h264dec). That however, required vpu_v4l2 to be enabled in the device tree.

[romelitzs]

I also don't understand your pipeline - why do you first decode h.264 stream just to re-encode it (and in separate branches)? This doesn't seem very efficient way of operating.

[VladanJovanovic]

Hi

what SW version are you using?

Can you try with v4l2h264dec and v4l2h264enc plugins instead of the vpudec/vpuenc ones?

I have no issues running below pipelines on 5.15.71 release from NXP on i.MX 8MP EVK:

root@imx8mpevk:~# gst-launch-1.0 filesrc location=btox_1080p_11_mbps.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! tee name=t ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! h264parse ! rtph264pay ! rtpbin ! udpsink t. ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! appsink

root@imx8mpevk:~# gst-launch-1.0 filesrc location=btox_1080p_11_mbps.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! tee name=t t. ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! autovideosink t. ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! fakesink

Best regards,

Vladan

[rradman_zenitel]

Hi Vladan, thank you for responding!

Could you please elaborate why is this pipeline not working with vpudec?

It is recommended in the documentation, I'll post a screenshot bellow.

Thank you,

Robert

[VladanJovanovic]

Hi @rradman_zenitel 

I'm not sure that vpudec doesn't work. I can start following pipeline through gst-launch without issues when I replace v4l2h264dec/enc with vpudec/enc:

root@imx8mpevk:~# gst-launch-1.0 filesrc location=btox_1080p_11_mbps.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! tee name=t ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! h264parse ! rtph264pay ! rtpbin ! udpsink t. ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! appsink

So issue might be coming from your app as well. There's no reason tee should cause problems as it's fully SW plugin.

I also don't understand your pipeline - why do you first decode h.264 stream just to re-encode it (and in separate branches)? This doesn't seem very efficient way of operating.

br,
Vladan

[rradman_zenitel]

Hi @VladanJovanovic 

We are decoding the h264 before tee element for the purpose of modifying it independently in each branch as needed (resolution change), and then encoding back to h264. For example, we can stream 1080p over one branch, 720p over the second, and so on. If there is a better (more efficient) way of doing so, please let me know, I would appreciate it.

Note: I've removed the resolution change from branches to make the debugging easier.

I've tried running the pipeline you recommended, but with vpudec and vpuenc_h264 :

gst-launch-1.0 filesrc location=movie.mp4 ! qtdemux !  h264parse ! vpudec ! tee name=t ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! h264parse ! rtph264pay ! rtpbin ! udpsink t. ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! appsink

Problem is, it crashes. I will post the logs in filesrc_logs.txt file.

Note2: Here is an example of a pipeline using the same mp4 file, that works (I have a display connected): gst-launch-1.0 filesrc location=movie.mp4 ! qtdemux ! h264parse ! vpudec ! waylandsink

BR

Robert

[VladanJovanovic]

It seems there were some issues on older versions of VPU driver and gstreamer. Please check this for more info:

https://community.nxp.com/t5/i-MX-Processors/MPEG2-PS-to-H-264-transcoding-failed-with-vpuenc-h264-o...

[VladanJovanovic]

What kernel and gstreamer versions are used for this?

Vladan

[rradman_zenitel]

Hi, here is the info you've asked for:

Kernel version: 5.15.60-imx8mp

Gstreamer version: 1.20

vpudec version: 4.7.1

Note: Uploading manifest to provide more visibility.

[VladanJovanovic]

Still don't see where the kernel is coming from.

Just to clarify - linux-imx kernels are released and supported by NXP. This is also true for other packages (gstreamer plugins etc.)

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

Version suggests it is not linux-imx kernel. Maybe linux-fslc based? In that case support should come from community.

Can you try NXP release? Especially if the issue reproduces on i.MX 8MP EVK? What HW are you testing this on?

One possible issue could also be lack of memory - e.g. not enough CMA memory allocated. How does /proc/meminfo look like on your system?

br,
Vladan

[rradman_zenitel]

Hi Vladan,

Our board is a proprietary hardware with Variscite SoM.

linux-variscite - https://github.com/varigit/linux-imx/tree/5.15-2.0.x-imx_var01

I've tested it with larger CMA but that didn't help. If you have any other ideas please share, thanks.

I've tried a different decoder element imxvpudec_h264. It is a part of gstreamer-imx plugins developed and supported by community, more info can be found there: https://github.com/Freescale/gstreamer-imx/.

Using imxvpudec_h264 seems to fix our problem - pipeline works fine, no crashing (still need to run some other tests to be sure).

Do you have some more information when comparing the three mentioned decoders - vpudec, v4l2h264dec, and imxvpudec_h264 ? Are they all safe to use? What are some of the advantages or disadvantages between them?

Best regards,

Robert

[VladanJovanovic]

I'd suggest to check with Variscite about the specific issue you're seeing. Based on gstreamer log it might be VPU kernel driver or VPU lib issue (especially if you also have same issue with v4l2h264dec), which is not happening on NXP release.

vpudec was initial decoder for i.MX 8M family of devices. While it is still enabled for supporting legacy applications, new development is happening on v4l2h264dec so that one is recommended for going forward.

As you mentioned as well, imxvpudec_h264 is coming from community. Not familiar with it and how well it performs, so can't really comment.

br,
Vladan

[NXPuser237]

Hey. 

Where is that v4l2h264dec element from

We seem to be using bitbake recepie from freescale

https://github.com/Freescale/meta-freescale/blob/kirkstone/recipes-multimedia/gstreamer/gstreamer1.0...


Point to somewhere around here
https://github.com/nxp-imx/gst-plugins-good/blob/MM_04.07.02_2210_L5.15.y/sys/v4l2/gstv4l2.c#L247

In general we cant seem to find v4l2dec. Is it on some other branch? Some other plugin set?
Can see that there is some conditional for registering v4l2 decoder plugin, could this maybe suggest there there is some requirement, secret, dependency for plugin to be enabled? https://github.com/nxp-imx/gst-plugins-good/blob/MM_04.07.02_2210_L5.15.y/sys/v4l2/gstv4l2.c#L177

[rradman_zenitel]

Thank you for your help!

Just one last thing, where can I find the package that holds the new decoder v4l2h264dec? Can I even install it or is it only available on NXP releases?

BR

Robert

[NXPuser237]

Very interesting topic. But we can see that if .tee is made before decoder then it works without crash or resolution drop. But then we are decoding twice. Did anyone figure out why it crashes when adding second branch using this ? 

v4l2src ! vpudec ! tee name=t

[Bio_TICFSL]

Hello,

Tee is not included in gstreamer 1.0, I guess this goes all by CPU.

Regards

[rradman_zenitel]

Hi, I'm not sure I understand what you're saying. Please try to elaborate with more detail if you can, thank you!

Note: I've added logs in the attachments.

BR

Robert