i.MX8MMini crop images before encoding with vpuenc_h264

[vincentz63]

I need to crop the images acquired from a CSI sensor before encoding them with vpuenc_h264.

The doc states that the Hentro encoder (H1) has a pre-processing stage able to do cropping.

How do I use this block from GStreamer?

I have tried the following to no avail:

GST_DEBUG=3 gst-launch-1.0 videotestsrc ! video/x-raw,width=1920,height=1080,format=YUY2 ! videocrop top=0 left=0 right=1000 bottom=500 ! vpuenc_h264 ! fakesink
0:00:00.045524419 4163 0x146c7920 ERROR GST_PIPELINE grammar.y:740:gst_parse_perform_link: could not link videocrop0 to vpuenc_h264-0

Looking at the git repo, commit #056d64fe seems to indicate cropping is supported ("MMFMWK-8718 Enable cropping for hantro encoder at preprocess stage"). Is this correct?

I have also tried using imxvideoconvert_g2d as a workaround, but I have hit a problem. I am not sure if I am using the element properly or not, so any example would be appreciated.

Here is the pipeline I use for my test:

gst-launch-1.0 videotestsrc ! video/x-raw,width=1920,height=1080,format=YUY2 ! videocrop top=0 left=0 right=1856 bottom=1016 ! imxvideoconvert_g2d ! fakesink

So long as right<=1856 and bottom<=1016, the pipeline negotiates and seems to work. However, if I increase 'right' or 'bottom' by just 1 pixel, it fails to negotiate with the following error.

Any idea what I might be doing wrong? Am I using imxvideoconvert_g2d in the right way?

gstutils.c:2053:gst_element_link_pads_full: no link possible from videocrop0 to imxvideoconvert_g2d0
0:00:00.259336221 4136 0x38d63c00 INFO default gstutils.c:2162:gst_element_link_pads_filtered: Could not link pads: videocrop0:(null) - imxvideoconvert_g2d0:(null)
0:00:00.259358847 4136 0x38d63c00 ERROR GST_PIPELINE grammar.y:740:gst_parse_perform_link: could not link videocrop0 to imxvideoconvert_g2d0

More error output below

multiview-mode=(string){ mono, left, right }
0:00:00.259080463 4136 0x38d63c00 DEBUG GST_CAPS gstpad.c:2280:gst_pad_link_check_compatible_unlocked:<imxvideoconvert_g2d0:sink> sink caps video/x-raw, width=(int)[ 64, 2147483647 ], height=(int)[ 64, 2147483647 ], format=(string){ RGB16, RGBx, RGBA, BGRA, BGRx, BGR16, ARGB, ABGR, xRGB, xBGR, I420, NV12, UYVY, YUY2, YVYU, YV12, NV16, NV21 }; video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComposition), width=(int)[ 64, 2147483647 ], height=(int)[ 64, 2147483647 ], format=(string){ RGB16, RGBx, RGBA, BGRA, BGRx, BGR16, ARGB, ABGR, xRGB, xBGR, I420, NV12, UYVY, YUY2, YVYU, YV12, NV16, NV21 }
0:00:00.259118714 4136 0x38d63c00 DEBUG GST_CAPS gstpad.c:2298:gst_pad_link_check_compatible_unlocked: caps are not compatible
0:00:00.259132715 4136 0x38d63c00 INFO GST_PADS gstpad.c:2434:gst_pad_link_prepare: caps are incompatible
0:00:00.259152590 4136 0x38d63c00 INFO GST_PADS gstpad.c:2529:gst_pad_link_full: link between videocrop0:src and imxvideoconvert_g2d0:sink failed: no common format
0:00:00.259199842 4136 0x38d63c00 DEBUG GST_BUS gstbus.c:316:gst_bus_post:<bus0> [msg 0x38eee090] posting on bus structure-change message: 0x38eee090, time 99:99:99.999999999, seq-num 11, element 'sink', GstMessageStructureChange, type=(GstStructureChangeType)GST_STRUCTURE_CHANGE_TYPE_PAD_LINK, owner=(GstElement)"\(GstVideoCrop\)\ videocrop0", busy=(boolean)false;
0:00:00.259222217 4136 0x38d63c00 DEBUG bin gstbin.c:3716:gst_bin_handle_message_func:<pipeline0> [msg 0x38eee090] handling child sink message of type structure-change
0:00:00.259244218 4136 0x38d63c00 DEBUG bin gstbin.c:958:bin_remove_messages:<imxvideoconvert_g2d0:sink> deleting message 0x38eee110 of type structure-change (types 0x00001000)
0:00:00.259266094 4136 0x38d63c00 DEBUG GST_BUS gstbus.c:351:gst_bus_post:<bus0> [msg 0x38eee090] dropped
0:00:00.259285344 4136 0x38d63c00 DEBUG GST_ELEMENT_PADS gstutils.c:1947:gst_element_link_pads_full: trying dest pad imxvideoconvert_g2d0:src
0:00:00.259299970 4136 0x38d63c00 DEBUG GST_ELEMENT_PADS gstutils.c:1999:gst_element_link_pads_full: we might have request pads on both sides, checking...
0:00:00.259317595 4136 0x38d63c00 DEBUG GST_ELEMENT_PADS gstutils.c:2053:gst_element_link_pads_full: no link possible from videocrop0 to imxvideoconvert_g2d0
0:00:00.259336221 4136 0x38d63c00 INFO default gstutils.c:2162:gst_element_link_pads_filtered: Could not link pads: videocrop0:(null) - imxvideoconvert_g2d0:(null)
0:00:00.259358847 4136 0x38d63c00 ERROR GST_PIPELINE grammar.y:740:gst_parse_perform_link: could not link videocrop0 to imxvideoconvert_g2d0
0:00:00.259407598 4136 0x38d63c00 INFO GST_PIPELINE grammar.y:652:gst_parse_perform_link: linking some pad of imxvideoconvert_g2d named imxvideoconvert_g2d0 to some pad of GstFakeSink named fakesink0 (0/0) with caps "(NULL)"
0:00:00.259428349 4136 0x38d63c00 INFO GST_ELEMENT_PADS gstutils.c:1774:gst_element_link_pads_full: trying to link element imxvideoconvert_g2d0:(any) to element fakesink0:(any)
0:00:00.259443474 4136 0x38d63c00 DEBUG GST_ELEMENT_PADS gstutils.c:1881:gst_element_link_pads_full: looping through allowed src and dest pads
0:00:00.259459725 4136 0x38d63c00 DEBUG GST_ELEMENT_PADS gstutils.c:1884:gst_element_link_pads_full: trying src pad imxvideoconvert_g2d0:sink
0:00:00.259475975 4136 0x38d63c00 DEBUG GST_ELEMENT_PADS gstutils.c:1884:gst_element_link_pads_full: trying src pad imxvideoconvert_g2d0:src
0:00:00.259493851 4136 0x38d63c00 DEBUG GST_ELEMENT_PADS gstutils.c:1105:gst_element_get_compatible_pad: finding pad in fakesink0 compatible with imxvideoconvert_g2d0:src
0:00:00.259517476 4136 0x38d63c00 INFO GST_PADS gstutils.c:1035:gst_pad_check_link: trying to link imxvideoconvert_g2d0:src and fakesink0:sink
0:00:00.259535602 4136 0x38d63c00 DEBUG GST_CAPS gstutils.c:3058:gst_pad_query_caps:<imxvideoconvert_g2d0:src> get pad caps with filter (NULL)
0:00:00.259553853 4136 0x38d63c00 DEBUG query gstquery.c:679:gst_query_new_custom: creating new query 0x38dfcde0 caps
0:00:00.259582978 4136 0x38d63c00 DEBUG GST_PADS gstpad.c:4053:gst_pad_query:<imxvideoconvert_g2d0:src> doing query 0x38dfcde0 (caps)
0:00:00.259601854 4136 0x38d63c00 DEBUG basetransform gstbasetransform.c:668:gst_base_transform_query_caps:<imxvideoconvert_g2d0:src> peer filter caps (NULL)

[vincentz63]

Hi Zhiming,

As I explained in my opening post, I want to use the H1 video encoder's cropping facility advertised in IMX8MMRM_Reference_Manual_REV2.pdf page 4806.

I have managed to use imxvideoconvert_g2d to crop the image, but using a GPU to crop when the encoder has this functionality built-in is a suboptimal workaround that increases power consumption and adds latency to the video pipeline.

Please re-read my last post where I explain that I have managed to crop the video using the H1 encoder, but the CPU usage shoots up to 60% on a dual core (1.5GHz) i.MX8MMini.

Could you please try to find information about the expected performance of the cropping facility of the H1 encoder? None of the NXP documents I have access to provide this information. Failing that, do you have an example or test I could run to check the expected performance?

Kind regards,

JP

[Zhiming_Liu]

Hi @vincentz63 

I didn't find performance test file in internal

BR

Zhiming

[vincentz63]

Hello,

Any hope of getting a more in-depth answer to my query?

Thanks,

JP

[vincentz63]

Hello,

Thanks for the quick response, but I'm still not clear how to go about using the cropping block of the H1 encoder in the i.MX8MMini.

i.MX_Linux_User's_Guide.pdf does not mention cropping anywhere but cropping has been implemented in the GStreamer element (see below).  Are are you saying that this code does not work?

The IMX8MMRM_Reference_Manual_REV2.pdf states that the vpuenc_h264 has support for cropping on page 4806, figure 15-14 (VPH H1 Encoder Block Diagram). The Ref Manual does not provide any details about it. I have traced the code from the GStreamer element to the H1 driver, but the H1 encoder used in the i.MX8MMini is an IP sold by VeriSilicon, and I do not have access to its datasheet, so understanding the driver is very difficult. 

I have managed to crop the video using the following pipeline (I had previously mis-interpreted the properties of videocrop).

v4l2src ! video/x-raw,width=1920,height=1080,format=YUY2 ! videocrop top=0 left=0 right=128 bottom=0 ! vpuenc_h264 ! rtph264pay name=pay0

It does crop 128 pixels to the right of the image, but the CPU usage shoots-up to 60% on a dual core i.MX. This is unlikely to be the expected performance of a hardware pre-processing block. So I suspect the driver memcpies the input image. But without the H1 datasheet, I do not know what to expect... although cropping should be implemented using a stride when reading the data, so no memcpy should be needed...

Looking at the source code, I can see that the metadata set by videocrop is used by the vpuenc_h264 GStreamer element:

plugins/vpu/gstvpuenc.c line 1022:

cropmeta = gst_buffer_get_video_crop_meta (frame->input_buffer);
if (cropmeta) {
enc->open_param.nPicWidth = cropmeta->width;
enc->open_param.nPicHeight = cropmeta->height;
}

and then later, line 1143:

enc_enc_param.nPicWidth = enc->open_param.nPicWidth;
enc_enc_param.nPicHeight = enc->open_param.nPicHeight;

I have verified that the memory used to store the input image is physical continuous memory so no memcpy occurs in the gstvpuenc.c (line 1063).

I have  traced those settings all the way to the VeriSilicon driver files (h264encapi.h/c, encpreprocess.c) but I cannot see how the cropping is done there.

So could you please explain to me how can I use the cropping block of the encoder, either via GStreamer or by pointing me to an example demonstrating its usage?

Thanks,

JP

[Zhiming_Liu]

Hi

You can see this issue to use g2d :https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/issues/843

BR

Zhiming

[Zhiming_Liu]

Hi

All supported operations of GST are located in 7.3 Multimedia use cases, i.MX_Linux_User's_Guide.pdf

BR

Zhiming