IMX6/8 Video chain with overlay posibilities in hardware performance

[TerryBarnaby1]

We are considering the use of a iMX6DL or iMX8M-Plus (when it is available) for a camera based inspection system we are designing. The video pipeline would be:

1. 1920x1080p30 video via MIPI-CSI

2. Overlay some text over the video

3. Tee to h264 encoder, package in MP4, store to MicroSd.

4. Tee to LVDS 1280x800 display and/or HDMI display

From what I can see 9and with some minor tests) an iMX6DL should be able to do most of this in hardware with little CPU usage. However I am unclear as if it can perform the 2. text overlay on the incoming video stream in hardware.

Any ideas if this is possible on an iMX6DL ?

Terry

[TerryBarnaby1]

We can use GStreamer or raw 'C' code to access the hardware as needed. Final output would be a LVDS 1280x800 and/or HDMI 1920x1080 display.

We would prefer to use the newer iMX8 chips, but I think the ones with the features we need are not available yet. The iMX8M-Plus or iMX8X industrial. The core features we need are:

1. MIPI-CSIcamera @ 1920x1080p30 input.

2. Overlay video with graphics (alpha blend).

3. H264 encode to a file while displaying on a LVDS 1280x800 display and/or a HDMI 1920x1080 display.

If there are any NXP SOC's available to us that can do that (in 100 -2000 off) we would be interested to hear.

[Bio_TICFSL]

Hi,

You can take the MX8MPlus

https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-proces....

[TerryBarnaby1]

But that appears to be pre-production and no distributers have it in stock as far as I am aware and there are no dates/times for delivery or no prices.

There are also no hardware datasheets and reference manuals available on the NXP website for the iMX8M-Plus as far as I can see.

[vincentz63]

Hello,

Did you find a solution to do alpha blending / text overlay on live video on any of those SoCs? If so would you be able to share what you did?

I ended-up implementing my own text overlay for the i.MX8M Mini a few years ago, but this is limited in the font it can use. I am now looking for a solution that would allow me to use anti-aliased font, ideally TextOverlay from GStreamer.

Thanks

[TerryBarnaby1]

Yes, although it was a bit complicated for my usage on an IMX8MP. I am using the QT5 GUI widget set with gstreamer. I used the gstreamer imxcompositor_g2d that uses the 2D graphics engine hardware in an IMX8MP to perform the overlay on the video stream of a slow frame rate overlay stream produced from an appsrc.

Very basically in Gstreamer (There is lots more in my usage):

"imxcompositor_g2d name=c sink_1::alpha=1.0 !  waylandsink name=\"videoSink\""

"videotestsrc pattern=smpte ! c.sink_1"

"appsrc name=\"appsrc\" ! videoconvert ! c.sink_1"

Then I used Qt5 to generate an overlay in a frame sized and pixel formated QImage using normal QT5 drawing so anything that Qt5 can draw (any fonts, graphics etc) will be overlayed.

On the GStreamer callback for the appsrc frame I then effectively did a memcpy from the QImage to the gstreamer buffer:

buffer = gst_buffer_new_wrapped_full((GstMemoryFlags)0, (gpointer)owOverlayImageDisplay->bits(), owOverlayImageDisplay->sizeInBytes(), 0, owOverlayImageDisplay->sizeInBytes(), NULL, NULL);

The appsrc and QImage was configured as needed for this.

The result was a QImage being overlayed at 4 frames per second and with relatively low CPU usage.

[vincentz63]

Hi Terry,

Thanks for the reply. What was the resolution of the video stream in your app? My use case is different from yours as I'm encoding the output of imxcompositor_g2d  with vpuenc_h264. It seems to keep-up at 1280p60, but drops a lot of frames at 1080p30.

[TerryBarnaby1]

The video stream was 1920x1080p25 with the overlay 1920x1080p4. I am also using vpuenc_h264 to encode the stream in my pipeline and saving to a MP4 file while displaying on an LCD panel as well has having some switchable T's to take snapshots etc.

On our custom IMX8MP hardware system performing the above takes around 25% over overall CPU usage (25% of 4 cores) with no missing frame issues etc.

[Bio_TICFSL]

Hello Terry,

Yes, it is possible, I understand that only V4L2 it is the only implemented sink that we have in the i.MX6 BSP . are you using Gstreamer? or what are you using or want to use to play the video. for the 1080p is better to use the MX8M.

Regards

[TerryBarnaby1]

Can you provide me with any information on how to perform a 1920x1080p30 video alpha composition with GUI graphics while sending to both a display and through h264 encoding to a file with an iMX8M platform (such as an iMX8M-Plus, iMX8DualXPlus or iMX8M-mini (ie iMX8's with VPU) ? At the moment I am trying to decide if an iMX8 SOC will suit our processing needs so I want to test this. I can use gstreamer, QT5, OpenCl, raw C++ I don't mind, at the moment I just need to see that the hardware is capable without saturating a single CPU core (ie. as much in hardware as possible). The sort of thing I want todo using a gstreamer example is:

gst-launch-1.0 compositor name=c sink_1::alpha=0.5 ! tee name=t ! queue ! autovideosink \
t. ! queue ! v4l2h264enc ! avimux ! filesink location=temp.avi \
v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! c.sink_0 \
multifilesrc location=./test1.png caps=image/png,framerate=1/1 ! pngdec ! imagefreeze ! c.sink_1

I have been playing with iMX6 and iMX8 platforms, but as soon as I try to use any form of graphics composer/mixer/overlay gstreamer element CPU usage goes through the roof. Even the iMX6's old imxcompositor_ipu using its IPU has problems. On the iMX8, having no IPU, I assume that the GPU will be needed for this. glvideomixer seems to work well when its direct output sync is a glimagesink but as soon as you want to send the output elsewhere again CPU usage is too high.

So are there any examples of performing the alpha composition on the GPU and then sending the frames to both the hardware h264 encoder and display with low CPU usage somewhere or info on how this can be done, if possible ?