imx8mp problems with vpudec and tee

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

imx8mp problems with vpudec and tee

Jump to solution
4,738 Views
rradman_zenitel
Contributor II

Hi,

 

I'm trying to create a GStreamer pipeline that decodes H264 video using vpudec and tee elements looking something like this:

 

 

 

v4l2src ! vpudec ! tee name=t

t. ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! h264parse ! rtph264pay ! rtpbin ! udpsink
t. ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! appsink

 

 

 

If I run the pipeline with this configuration while attaching only one of the branches to the tee elements, it works fine. As soon as I add the second branch to the tee element (ordering doesn't matter), my program crashes. I've tried adding queue elements with max-size-time=0 property to the first pipeline (before or after the vpudec, or both) but It did nothing.

I've managed to prevent the crashing by adding more elements to the first pipeline so it would look like this:

 

 

 

v4l2src ! vpudec ! queue ! videorate ! imxvideoconvert_g2d ! tee

 

 

 

By doing so, a new problem appears - Adding a second branch with lower video resolution affects the video resolution on the first branch.

Steps to reproduce:
- add branch 1 at 1080p

- add branch 2 at 240p

Result: Branch 1 resolution drops significantly. Looks like 240p.

This happens on-the-fly, without restarting the pipelines.

 

Note: When running the similar configuration with RAW camera on the same device (no decoding needed), there is no any issues.

 

 

 

v4l2src ! tee

 

 

EDIT: I've added logs as an attachment.

Pipeline used:

 

GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-h264 ! vpudec ! tee name=t 
t. ! queue ! videorate ! imxvideoconvert_g2d ! videoconvert ! vpuenc_h264 ! appsink 
t. ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! h264parse ! rtph264pay ! udpsink 

 

Individually both branches work fine, program crashes when adding second branch.

 

Every advice is much appreciated.

If needed I can provide more information.

Thanks!

Robert

 

Labels (1)
0 Kudos
1 Solution
3,752 Views
rradman_zenitel
Contributor II

Issue was fixed by installing the new decoder (v4l2h264dec). That however, required vpu_v4l2 to be enabled in the device tree.

View solution in original post

0 Kudos
17 Replies
3,753 Views
rradman_zenitel
Contributor II

Issue was fixed by installing the new decoder (v4l2h264dec). That however, required vpu_v4l2 to be enabled in the device tree.

0 Kudos
4,444 Views
romelitzs
Contributor I

I also don't understand your pipeline - why do you first decode h.264 stream just to re-encode it (and in separate branches)? This doesn't seem very efficient way of operating.

 

0 Kudos
4,588 Views
VladanJovanovic
NXP Employee
NXP Employee

Hi

what SW version are you using?

Can you try with v4l2h264dec and v4l2h264enc plugins instead of the vpudec/vpuenc ones?

I have no issues running below pipelines on 5.15.71 release from NXP on i.MX 8MP EVK:

root@imx8mpevk:~# gst-launch-1.0 filesrc location=btox_1080p_11_mbps.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! tee name=t ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! h264parse ! rtph264pay ! rtpbin ! udpsink t. ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! appsink

root@imx8mpevk:~# gst-launch-1.0 filesrc location=btox_1080p_11_mbps.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! tee name=t t. ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! autovideosink t. ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! fakesink

Best regards,

Vladan

0 Kudos
4,559 Views
rradman_zenitel
Contributor II

Hi Vladan, thank you for responding!

Could you please elaborate why is this pipeline not working with vpudec?

It is recommended in the documentation, I'll post a screenshot bellow.

rradman_zenitel_0-1675675508830.png

Thank you,

Robert

0 Kudos
4,546 Views
VladanJovanovic
NXP Employee
NXP Employee

Hi @rradman_zenitel 

I'm not sure that vpudec doesn't work. I can start following pipeline through gst-launch without issues when I replace v4l2h264dec/enc with vpudec/enc:

root@imx8mpevk:~# gst-launch-1.0 filesrc location=btox_1080p_11_mbps.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! tee name=t ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! h264parse ! rtph264pay ! rtpbin ! udpsink t. ! queue ! videorate ! imxvideoconvert_g2d ! v4l2h264enc ! appsink

So issue might be coming from your app as well. There's no reason tee should cause problems as it's fully SW plugin.

I also don't understand your pipeline - why do you first decode h.264 stream just to re-encode it (and in separate branches)? This doesn't seem very efficient way of operating.

br,
Vladan

 

0 Kudos
4,535 Views
rradman_zenitel
Contributor II

Hi @VladanJovanovic 

We are decoding the h264 before tee element for the purpose of modifying it independently in each branch as needed (resolution change), and then encoding back to h264. For example, we can stream 1080p over one branch, 720p over the second, and so on. If there is a better (more efficient) way of doing so, please let me know, I would appreciate it.

Note: I've removed the resolution change from branches to make the debugging easier.

 

I've tried running the pipeline you recommended, but with vpudec and vpuenc_h264 :

gst-launch-1.0 filesrc location=movie.mp4 ! qtdemux !  h264parse ! vpudec ! tee name=t ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! h264parse ! rtph264pay ! rtpbin ! udpsink t. ! queue ! videorate ! imxvideoconvert_g2d ! vpuenc_h264 ! appsink

Problem is, it crashes. I will post the logs in filesrc_logs.txt file.

Note2: Here is an example of a pipeline using the same mp4 file, that works (I have a display connected): gst-launch-1.0 filesrc location=movie.mp4 ! qtdemux ! h264parse ! vpudec ! waylandsink

 

BR

Robert

 

0 Kudos
4,505 Views
VladanJovanovic
NXP Employee
NXP Employee

It seems there were some issues on older versions of VPU driver and gstreamer. Please check this for more info:

https://community.nxp.com/t5/i-MX-Processors/MPEG2-PS-to-H-264-transcoding-failed-with-vpuenc-h264-o...

 

 

0 Kudos
4,512 Views
VladanJovanovic
NXP Employee
NXP Employee

What kernel and gstreamer versions are used for this?

Vladan

 

0 Kudos
4,499 Views
rradman_zenitel
Contributor II

Hi, here is the info you've asked for:

Kernel version: 5.15.60-imx8mp

Gstreamer version: 1.20

vpudec version: 4.7.1

Note: Uploading manifest to provide more visibility.

0 Kudos
4,480 Views
VladanJovanovic
NXP Employee
NXP Employee

Still don't see where the kernel is coming from.

Just to clarify - linux-imx kernels are released and supported by NXP. This is also true for other packages (gstreamer plugins etc.)

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

Version suggests it is not linux-imx kernel. Maybe linux-fslc based? In that case support should come from community.

Can you try NXP release? Especially if the issue reproduces on i.MX 8MP EVK? What HW are you testing this on?

One possible issue could also be lack of memory - e.g. not enough CMA memory allocated. How does /proc/meminfo look like on your system?

br,
Vladan

 

 

0 Kudos
4,462 Views
rradman_zenitel
Contributor II

Hi Vladan,

Our board is a proprietary hardware with Variscite SoM.

linux-variscite - https://github.com/varigit/linux-imx/tree/5.15-2.0.x-imx_var01

I've tested it with larger CMA but that didn't help. If you have any other ideas please share, thanks.

I've tried a different decoder element imxvpudec_h264. It is a part of gstreamer-imx plugins developed and supported by community, more info can be found there: https://github.com/Freescale/gstreamer-imx/.

Using imxvpudec_h264 seems to fix our problem - pipeline works fine, no crashing (still need to run some other tests to be sure).

Do you have some more information when comparing the three mentioned decoders - vpudec, v4l2h264dec, and imxvpudec_h264 ? Are they all safe to use? What are some of the advantages or disadvantages between them?

Best regards,

Robert

0 Kudos
4,455 Views
VladanJovanovic
NXP Employee
NXP Employee

I'd suggest to check with Variscite about the specific issue you're seeing. Based on gstreamer log it might be VPU kernel driver or VPU lib issue (especially if you also have same issue with v4l2h264dec), which is not happening on NXP release.

vpudec was initial decoder for i.MX 8M family of devices. While it is still enabled for supporting legacy applications, new development is happening on v4l2h264dec so that one is recommended for going forward.

As you mentioned as well, imxvpudec_h264 is coming from community. Not familiar with it and how well it performs, so can't really comment.

br,
Vladan

 

 

0 Kudos
4,409 Views
NXPuser237
Contributor I

Hey. 

Where is that v4l2h264dec element from

 

We seem to be using bitbake recepie from freescale

https://github.com/Freescale/meta-freescale/blob/kirkstone/recipes-multimedia/gstreamer/gstreamer1.0...


Point to somewhere around here
https://github.com/nxp-imx/gst-plugins-good/blob/MM_04.07.02_2210_L5.15.y/sys/v4l2/gstv4l2.c#L247

In general we cant seem to find v4l2dec. Is it on some other branch? Some other plugin set?
Can see that there is some conditional for registering v4l2 decoder plugin, could this maybe suggest there there is some requirement, secret, dependency for plugin to be enabled? https://github.com/nxp-imx/gst-plugins-good/blob/MM_04.07.02_2210_L5.15.y/sys/v4l2/gstv4l2.c#L177

0 Kudos
4,419 Views
rradman_zenitel
Contributor II

Thank you for your help!

Just one last thing, where can I find the package that holds the new decoder v4l2h264dec? Can I even install it or is it only available on NXP releases?

BR

Robert

0 Kudos
4,637 Views
NXPuser237
Contributor I

Very interesting topic. But we can see that if .tee is made before decoder then it works without crash or resolution drop. But then we are decoding twice. Did anyone figure out why it crashes when adding second branch using this ? 

v4l2src ! vpudec ! tee name=t

 

0 Kudos
4,706 Views
Bio_TICFSL
NXP TechSupport
NXP TechSupport

Hello,

Tee is not included in gstreamer 1.0, I guess this goes all by CPU.

 

Regards

0 Kudos
4,684 Views
rradman_zenitel
Contributor II

Hi, I'm not sure I understand what you're saying. Please try to elaborate with more detail if you can, thank you!

Note: I've added logs in the attachments.

BR

Robert

0 Kudos