i.MX6DL not smooth full HD accelerated video play

[frantisekhacker]

Hello,

we would like to develop an equipment with GPU accelerated full-HD video H.264 blended with 0.5alpha still picture over it. No Android.
Questions:
a) Is this intended equipment possible to do with GPU acceleration?
b) Is this possible to do it with g2d?
c) Is v4l2 supported by NXP in future? How to resize the movie in 800x600 window? Please see 5 below.

Thank you,
FH

We use for testing:
1) i.MX6 Series SABRE for SmartDevices (MCIMX6DL-SDP) with LCD touch 1024x768
2) Linux imx6dlsabresd 4.1.15-2.1.0+g30278ab with compiled imx-gstreamer plugins
3)  /sys/class/graphics/fb0: DISP3 BG - DI1 U:1024x768p-60  ldb     2-layer-fb-bg
   /sys/class/graphics/fb1: DISP3 FG                       overlay 2-layer-fb-fg
   /sys/class/graphics/fb2: DISP3 BG       U:1920x1080p-60 hdmi    1-layer-fb

   - (to fb0) movie:         1920x765@23.976FPS(M4V codec (M4V /M4A /mp42/isom))
   - (to fb1) still picture: PNG image data, 1024 x 768, 8-bit/color RGBA, non-interlaced

   -   loc_alpha.alpha_in_pixel = 1;
     loc_alpha.enable = 1;
     fd = open("/dev/fb0", O_RDWR, 0);
     ioctl(fd, MXCFB_SET_LOC_ALPHA, &loc_alpha);
     ioctl(fd, MXCFB_WAIT_FOR_VSYNC, 0);
     ioctl(fd, FBIOBLANK, FB_BLANK_UNBLANK);

4) no HW changes, no SW changes, in Yocto added compile imx-gstreamer plugins
5) expected: alpha blending, smooth video play, no CPU at 100%
   observed: alpha blending is correct (picture and movie are mixed)
             background video play is visible, good colours and size, but not smooth play
             (=like movie on a powerless hardware without GPU acceleration)
             CPU is on 11%

             When we use v4l2:
             "gst-launch-1.0 filesrc location=N_movie.m4v ! qtdemux ! queue ! vpudec  ! imxv4l2sink device=/dev/video16"
             the movie play smoothly, but there is unfortunately no way to resize it inside 800x600 window

6)  echo 0 > /sys/class/graphics/fb1/blank
   echo 0 > /sys/class/graphics/fb0/blank
   set_alpha.bin
   WINDOW="window-x-coord=100 window-y-coord=100 window-width=800 window-height=600"
   gst-launch-1.0 filesrc location=2N_picture_05alpha.png ! pngdec ! imagefreeze ! queue ! imxg2dvideosink force-aspect-ratio=true framebuffer=/dev/fb1
   gst-launch-1.0 filesrc location=2N_movie.m4v typefind=true ! qtdemux ! queue ! vpudec ! imxg2dvideosink framebuffer=/dev/fb0 $WINDOW

7) always not smoothly video overplay
8) LCD touch 1024x768, like here

[igorpadykov]

Hi Frantisek

one can test video capabilities with Demo Image

https://www.nxp.com/webapp/Download?colCode=L4.1.15_2.0.0_MX6QDLSOLO&appType=license&location=null 

follow sect.7 Multimedia, sect.7.3.15 Video conversion attached Linux Guide

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[frantisekhacker]

Hi Igor,

thank you for the reply.

1. I have tested the older demo image L4.1.15_2.0.0 (instead of our newest L4.1.15_2.1.0):
   • there are no imxg2dxxx gstreamer plugins compiled in, so the G2D can not be tested here by gstreamer at all
   • yes, v4l2 also works fine here (smoothly) as in our L4.1.15_2.1.0

2. I realized (in L4.1.15_2.1.0) that if I use decodebin!videoconvert instead of qtdemux!vpudec, everythink works fine

   as expected, accelerated (smooth background video play, alpha blending image on top of it, resize, CPU at 20%):

gst-launch-1.0 filesrc location=2N_movie.m4v typefind=true ! qtdemux ! queue ! vpudec ! imxg2dvideosink framebuffer=/dev/fb0 $WINDOW

gst-launch-1.0 filesrc location=2N_movie.m4v typefind=true ! decodebin ! queue ! videoconvert ! imxg2dvideosink framebuffer=/dev/fb0 $WINDOW

   So my previous a), b) and c) questions are solved now.

   Can you please explain, what is here wrong with qtdemux!vpudec?

My next questions are:

1. Instead of blended still image on top of video we would like to use OpenGL draw functions aplha blended with background video. Is it possible? How to set the OpenGL to draw in framebuffer /dev/fb1? No Andorid, no X11,... but also accelerated.
2. Will NXP support g2d in the future? Is it preferred?

Thank you,

and best regards,

FH

[mtx512]

I think you may be mixing plugins from two different sources (which aren't compatible), the FSL gst-1.0 plugs are from here and include vpudec. There's also an open source implementation gstreamer-imx that includes imxg2dvideosink. The g2d api allow overlaying of multi buffers when blitting to the output surface however I'm not sure if this is implement by the gst-plugin (simple demo calling g2d api directly). Since gstreamer-imx source code is available then I guess worst case is that you will need to support it yourself. 

[frantisekhacker]

Thank you, answers after hours of investigation:

a) yes, this equiptment is possible to do with GPU acceleration - we made as Linux native app:

1. H.264 video decoded, cropped and resized to /dev/fb0, displayed thru G2D (NXP Yocto example mxc_vpu_test and its Android displaying)
2. setted GLES2.0 layer drawing to /dev/fb1 with alpha setted to 0.5 by ioctl

b) yes, G2D is working perfectly (in mxc_vpu_test example we reused codes written for Android to be used without Android)

c) resize is done by G2D also

FH