IMX8 Video overlay with graphics before H264 encode

[TerryBarnaby1]

Hi we are designing a visual inspection system using a 1920x1080p30 video camera.

We need to overlay graphics onto the video stream prior to H264 encode as well as to the display on a low power system. I am not finding any information on how to do this with a SOC like an iMX8M-Plus. The iMX6 had an IPU that could compose/overlay graphics onto the video stream although at a max resolution of 1024x1024 (could do this 4 x for 1920x1080). The iMX8M family seems to have lost this aspect of video processing.

So I guess the 2D or 3D GPU would have to do this composition/overlay to memory while also performing normal display work. It appears there are no gstreamer plugins to handle this so I would have to create something. However at this stage I want to know:

1. Would the iMX8-Plus hardware be capable of this with minimal CPU usage (camera would be YUV I422/I421 and composer/overlay output needs to be fed to the VPU encoder (NV12 ?) as well as to the display) ?

2. Any ideas on how to go about this ? We would have to use the GPU whilst the normal X11/Wayland graphics GUI is in use and feed the data through zero copy dma buffer pipeline etc.

Terry

[AlexxS]

@TerryBarnaby1 , what sort of framerate were you able to get using this solution? We're evaluating different options and there are very little resources on what potential solutions are to draw shapes or overlay text on frames captured from the camera.

Thanks!

[TerryBarnaby1]

Well we have an analogue camera feeding a 1920x1080p25 video stream and are overlaying on that. We create the overlay image using Qt into a QImage of the full 1920x1080 size and then use a gstreamer appSrc to inject this into one of the imxcompositor_g2d sources along with the video stream with alpha=1.0.

In our case we generate the appSrc gstreamer overlay frames at 5 fps to save CPU usage on the Qt drawing side as we don't need it any faster than this, but have done the full 25 fps without issue.

With our system which is capturing a FHD video stream, overlaying text/graphics, displaying on a LCD panel while H264 encoding and saving toi a file as well as managing a sound stream, the overal usage (4 cores) of the imx8mp is about 25%.

But working with the IMX8mp with gstreamer is a bit of a minefield. It is relatively complex system under the hood without detailed documentation and this means you often have to play with gstreamer arrangements and settings, quite a few which are not obvious, to get things working efficiently.

[AlexxS]

Thank you for the detailed answer! A few more questions based on this, if you don't mind:
1. How are you keeping the QT overlay and the video frames in sync? I presume the data overlay needs to be correctly synchronized to the video frame to which it corresponds to.
2. I'm not that experienced with the QT + Gstreamer combination, can you please give a few more details about how these two are interfaced? I found some information on how to get video using Gstreamer inside QT Apps, but not what we're trying to do.

PS: My initial interpretation of your post is that you're saving the overlay as an image and then injecting that using appSrc. That link is what's not that clear to me. 

Thank you!

[TerryBarnaby1]

Its simple in our application. The overlay QImage is created asynchronously about once per second (Actually two QImages, one being drawn, one for display with a Mutex for switching between).

The gstreamer appsrc element is configured to emit video frames at 5 fps and takes the QImage and injects it into the gstreamer stream via the imxcompositor_g2d.

ogstOverlaysrc=gst_bin_get_by_name(GST_BIN(ogstPipeline), "appsrc");
if(ogstOverlaySrc){
g_object_set(G_OBJECT(ogstOverlaySrc), "caps", gst_caps_new_simple("video/x-raw",
 "format", G_TYPE_STRING, "BGRA", "width", G_TYPE_INT, osize.width(),
 "height", G_TYPE_INT, osize.height(),
 "framerate", GST_TYPE_FRACTION, 5, 1,
 NULL), NULL);

g_object_set(G_OBJECT(ogstOverlaySrc), "stream-type", 0, // GST_APP_STREAM_TYPE_STREAM
 "format", GST_FORMAT_TIME, "is-live", FALSE, NULL);

g_signal_connect(ogstOverlaySrc, "need-data", G_CALLBACK(gstOverlayAddFrameCallback), gpointer(this));
}

 Pipeline bits:

imxcompositor_g2d name=c latency=20000000 sink_1::alpha=1.0 ! identity drop-allocation=true
v4l2src device=/dev/video3 ! video/x-raw,width=1920,height=1080,framerate=25/1 ! c.sink_0
appsrc name="appsrc" ! videoconvert ! video/x-raw,format=ARGB ! c.sink_1

[TerryBarnaby1]

Many thanks for the info. I would like to try this to prove that the output of this can be taken through the H264 VPU encoder and the overall CPU usage of this. However where and how would I get the gstreamer imxcompositor_g2d plugin ? I have been trying to build a suitable community or nxp based Yocto build for the iMX6 and iMX8 test platforms I have but have never seen this plugin ? The NXP gstreamer plugins seem to be in quite a mess at the moment.

Should the user level part of this be in the gstreamer1.0-plugins-imx package ?

If I build a IMX community Yocto release using: "https://github.com/Freescale/fsl-community-bsp-platform -b dunfell" this cannot build a gstreamer1.0-plugins-imx package.

If I build a IMX NXP Yocto release using: "https://source.codeaurora.org/external/imx/imx-manifest -b imx-linux-zeus -m imx-5.4.70-2.3.0.xml" for a imx8mqevk machine there is no imxcompositor_g2d plugin as far as I can seen in the build (cannot run it as I don't have a imx8mqevk) and if I add "gstreamer1.0-plugins-imx" to the build (IMAGE_INSTALL_append) I get the error "imx8mqevk (not in COMPATIBLE_MACHINE)".

So what should I build to get a suitable Yocto Linux system that will provide a working imxcompositor_g2d gstreamer plugin ?

[Zhiming_Liu]

Hi

The i.MX8MQ doesn't have 2D GPU,so you can not use imxcompositor_g2d,you can try another imxcompositor

BR

[TerryBarnaby1]

Thanks for the information, I had just worked out that the imx8m didn't support some video processing but I didn't release it didn't have a 2D GPU! This iMX8 range is very messy and has strange part naming!

However the basic question remains, we are not targeting the imx8mq for our design, we will probably be using a IMX8M-Plus or IMX8-DualXPlus. What I am trying to determine is how to build a suitable Yocto release with support for hardware video compositing before H264 in general.

The software support has changed a lot over the last 5 years and there are many routes to producing a Yocto release and many graphics and video processing driver changes in the kernel and outside etc. There doesn't appear to be a simple overall guide on these changes and where the direction is going and what user level API and what kernel and kernel level API's are needed. So I'm trying to understand this so we can first test or video processing concept and then design and board and produce a suitable Yocto build for this custom board.

For example am I right in thinking that the imx-gst1.0-plugin package has replaced the gstreamer1.0-plugins-imx package and when and what lower level API's/drivers does it need/support (DRI ?). I would like to build systems for iMX6 and iMX8M-Plus that are compatible but I'm not sure this is possible.

Terry

[Zhiming_Liu]

Hi

All bsp release note and the details about i.MX GStreamer 1.0 plugins  are in i.MX_Linux_Release_Notes.pdf (Multimedia)

i.MX_Linux_Users_Guide.pdf talks about how to do tests.

About the last point you mentioned, the release note has said:

i.MX 6 Family:
• overlaysink : G2D-based video sink plugin
• imxv4l2sink: V4L2-based video sink plugin

[TerryBarnaby1]

Thanks for the info, but that is specific information to a specific NXP BSP release.

I need to produce a Yocto release for a custom board and thus need to configure a Yocto build for this to get the most current and up-to-date video processing gstreamer elements. The NXP "Yocto" builds are different from the community builds (which are more mainline) and there is the BSP type and the three different kernel variants and then all of the paraphernalia of different kernel driver configurations etc. Aslo old Yocto builds have different gstreamer elements and drivers etc.

I am just trying to understand what the current/planned underlying video processing software tree is, so I follow this in our builds. It has been stated to use the community Yocto BSP for custom boards but this is quite different to the NXP Yocto wrt video processing. For example at the user level there is the gstreamer1.0-plugins-imx and imx-gst1.0-plugin package both of which do not build with a community based Yocto dunfell build for an IMX6DL.

Am I right in thinking that the imx-gst1.0-plugin package has taken over from the gstreamer1.0-plugins-imx for all IMX variants and does this rely on the DRI GPU interface route ?

I just want to understand the overall hardware video processing software/hardware architecture/packages and its history so I can develop in the most sensible way.

Terry

[Zhiming_Liu · Accepted solution]

Hi

Yes,imx-gst1.0-plugin has replaced the gstreamer1.0-plugins-imx.

In i.MX6 Family :Video render relys on G2D, not DRI

In i.MX8 Family:Video render relys on DRI,due to Wayland using DRI

[TerryBarnaby1]

Many thanks.

It would be nice to have an overview of the video processing architecture (hardware and software layers/API's and packages) and the history in the "i.MX Linux® User's Guide" and how this is supported in the NXP Yocto BSP and community Yocto BSP.

[Zhiming_Liu]

Hi

1.YUV422 and YUV420 input and output formats are also supported, supports encoder (NV12).

2.You can see 7.3.16 Video composition in i.MX_Linux_User's_Guide.pdf
imxcompositor_g2d uses corresponding hardware to accelerate video composition. It can be used to composite multiple
videos into one. The video position, size, and rotation can be specified while composition. Video color space conversion is
also performed automatically if input and output video are not same. Each video can be set to an alpha and z-order value to
get alpha blending and video blending sequence.

[NagendraB]

Hello 

I am working with similar requirements , I would like to understand how to test video overlays on imx8m plus platform, unfortunately the doc Linux user guide  dated L5.4.70_2.3.0, 13 January 2021 does not have imx8m-plus specific info and the plugin overlaysink is not available.

[TerryBarnaby1]

With the current (NXP Yocto hardknott) based Linux I have found you can to use the graphics 2D engine to perform the overlay processing. The gstreamed imxcompositor_g2d module can do this. A typical gstreamer command could be like:

gst-launch-1.0 imxcompositor_g2d name=c sink_1::alpha=0.5 ! waylandsink v4l2src device=/dev/video3 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! c.sink_0 multifilesrc location=./test1.png caps=image/png,framerate=1/1 ! pngdec ! imagefreeze ! c.sink_1

This will take a full HD camera stream and overlay the test1.png file on it. Following the imxcompositor_g2d you can pass this video stream through "vpuenc_h264 ! h264parse ! avimux ! filesink location=/tmp/temp.avi" to encode using the hardware h264 encoder possibly teeing it to the display as well as the h264 encoder if needed.

[NagendraB]

Hello Terry

Thanks for the inputs, If I want to blend video  layer with wayland GUI layer is it possible with similar approach ?

[TerryBarnaby1]

I guess that depends on exactly what you want to do. You will need to understand the iMX8mp hardware, the implemented gstreamer modules, the wayland server code, your GUI toolkit etc. info on which is lacking and I'm no expert. And it also depends on what you mean by Wayland GUI layer.

In my case I needed to overlay some text on the video stream prior to displaying it in a Qt window as well as h264 encoding it to a file and I am using the Qt GUI platform. In my code the graphical "layer" (which comes from test1.png in my example) that is overlaid on the video is generated in a Qt QImage using normal Qt drawing primitives and the gstreamer "appsrc name=\"appsrc\" ! videoconvert ! imagefreeze ! c.sink_1" partial pipeline is used to feed this into the imxcompositor_g2d. The screen output is "waylandsink name=\"videoSink\"" and there is some C++ GUI code to move and size this walyandsink area over a particular Qt window in my application.

[NagendraB]

hello Terry
In my case,  consider use case like simple wayland application with a video window inside it with a position and resolution configured , what could be the best approach ?should I use multiple egl surfaces ? or wayland sub-surfaces ? could you suggest which approach would be better?

[NagendraB]

Hello 

consider  simple wayland application which run multiple videos in it

And another point , I am trying to use alpha property of waylandsink (imx8m-plus) which is not working, any limitation here ?

[NagendraB]

hello NXP

the alpha property of waylandsink is not working (5.4.70- imx8mplus) , is there any limitation? 

[NagendraB]

hello 
Any inputs on my above query ? is waylandsink has limitation with respect to alpha property ?
Also any Idea how to use chromakey  property  using GPU/VPU ?