topic Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini in i.MX Processors

v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

vincentz63 — Fri, 22 Nov 2019 12:03:45 GMT

I am experimenting with imxcompositor_g2d in order to overlay some text in an RTP pipeline but as soon as it is linked together with the v4l2src, the v4l2src element drops frames and prints "lost frames detected".

I'd like to use the hardware acceleration for the overlay as the textoverlay element consumes too much CPU and ditto for the compositor element.

I have boiled down the problem to the following very simple GStreamer pipeline:

imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! c.sink_0

This is running on an i.MX8M Mini.

The CPU usage is pretty low, around 7%. Profiling the app shows that the following functions are using up most of the CPU:

__pi___inval_dcache_area
__pi___clean_dcache_area_poc
_raw_spin_unlock_irqrestore

And an example of the callstack is below (taken from perf).

I'm guessing I'm doing something seriously wrong here as otherwise the imxcompositor_g2d would be unusable.

If I replace the imxcompositor_g2d with the compositor element, the CPU usage goes to 80%, but the v4l2 does not report dropped frames.

84.01%     0.00%  comp:src        libglib-2.0.so.0.5400.3           [.] g_thread_proxy
            |
            ---g_thread_proxy
               g_thread_pool_thread_proxy
               default_func
               gst_task_func
               |          
                --83.78%--gst_aggregator_aggregate_func
                          |          
                          |--81.03%--gst_video_aggregator_aggregate
                          |          |          
                          |          |--72.52%--gst_imxcompositor_aggregate_frames
                          |          |          |          
                          |          |          |--36.69%--imx_g2d_fill_color
                          |          |          |          |          
                          |          |          |          |--33.24%--__GI___ioctl
                          |          |          |          |          |          
                          |          |          |          |           --33.16%--__sys_trace_return
                          |          |          |          |                     sys_ioctl
                          |          |          |          |                     |          
                          |          |          |          |                      --33.10%--do_vfs_ioctl
                          |          |          |          |                                dma_buf_ioctl
                          |          |          |          |                                |          
                          |          |          |          |                                |--13.44%--dma_buf_map_attachment
                          |          |          |          |                                |          ion_map_dma_buf
                          |          |          |          |                                |          __pi___clean_dcache_area_poc
                          |          |          |          |                                |          
                          |          |          |          |                                |--13.00%--dma_buf_unmap_attachment
                          |          |          |          |                                |          ion_unmap_dma_buf
                          |          |          |          |                                |          |          
                          |          |          |          |                                |           --12.93%--__pi___inval_dcache_area
                          |          |          |          |                                |          
                          |          |          |          |                                 --6.11%--arch_setup_dma_ops
                          |          |          |          |                                           |          
                          |          |          |          |                                            --6.04%--of_find_compatible_node
                          |          |          |          |                                                      _raw_spin_unlock_irqrestore

Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

Bio_TICFSL — Wed, 27 Nov 2019 15:26:09 GMT

Hello JP,

I am a little confused by your question. is you gstreamer gst-launch-1.0 command is

imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! c.sink_0?

This seems not a valid gstreamer command.

Could you provide your whole gstreamer command?

Regards

Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

vincentz63 — Wed, 27 Nov 2019 16:14:44 GMT

Hello,

Sorry I have to manually type the command and I missed the caps filter on the V4L2SRC element.

GST_DEBUG=3 gst-launch-1.0 "imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0"

It prints the following warning: "v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected".

Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

Bio_TICFSL — Fri, 06 Dec 2019 18:44:19 GMT

which BSP version are you using? I am using 4.14-sumo, I cannot run your command

ERROR: pipeline could not be constructed: no element "imxcompositor_g2d".

I tried both:

and

gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0

Regards

Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

vincentz63 — Mon, 09 Dec 2019 17:10:34 GMT

Hi,

You need to remove the quotes (")

So:

GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0

EDIT:

Forgot to sat that I'm using imx-yocto-L4.14.98_2.0.0_ga

Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

Bio_TICFSL — Wed, 11 Dec 2019 15:20:19 GMT

Yes, I tried this one, not working. There is no imxcompositor_g2d in bsp I am using. Which bsp are your using?

And which board? imx8m mini board?

Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

vincentz63 — Wed, 11 Dec 2019 15:37:41 GMT

It's for the i.MX8M Mini (as stated in the title of the question) and I am using the i.MX8MMini-EVK. The image is built for MACHINE=imx8mmevk and DISTRO=fsl-imx-xwayland. The image is fsl-image-validation-imx.

From memory, the element was present on the stock image present on the eMMC on the EVK.

The imxcompositor_g2d element is one of the elements provided by NXP.

It is somewhat documented in the "i.MX Linux User's Guide", page 50 "Video Composition": "imxcompositor_g2d uses corresponding hardware to accelerate video composition..."

Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

vincentz63 — Thu, 12 Dec 2019 10:47:51 GMT

Hi,

Any chance of getting some feedback soon?

To be clear I'm using L4.14.98_2.0.0_ga on the i.MX8MMini-EVK.

Thanks

Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

Bio_TICFSL — Thu, 09 Jan 2020 14:25:33 GMT

Hello

We found a multimedia BSP with the specific plugin:

Environment:

i.MX 8MM EVK

Camera OV5640

BSP L4.19.35_1.1.0 multimedia image

here is the log:

-------------------

GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! waylandsink sync=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0

Setting pipeline to PAUSED ...

0:00:00.056040250 4072 0x3afa4c00 WARN v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4

L2_EVENT_EOS event for device '/dev/video0'.

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

[ 196.831569] alloc_contig_range: [81e00, 821f5) PFNs busy

[ 196.862006] ov5640_mipi 2-003c: s_stream: 1

[ 197.061215] skip frame 1

0:00:01.123702000 4072 0x3aef3370 WARN v4l2bufferpool gstv4l2bufferpool.c:1291:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2

_buffer.field to ANY

0:00:01.125192500 4072 0x3aef32d0 WARN videopool gstvideopool.c:228:video_buffer_pool_set_config:<waylandpool0> Provided size is to small for the cap

s: 0 < 4147200

0:00:01.132573750 4072 0x3aef32d0 FIXME basesink gstbasesink.c:3248:gst_base_sink_default_event:<waylandsink0> stream-start event without group-id. C

onsider implementing group-id handling in the upstream elements

0:00:01.139302125 4072 0x3aef32d0 WARN waylandsink wlwindow.c:333:gst_wl_window_new_internal: init surface_state fail, fallback to scale=1 fullscreen (

1920x1048)

0:00:01.756389875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 16 - ts: 0:00:01.698179

750

0:00:02.057064125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 7 - ts: 0:00:01.9988748

0:00:02.190468875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:02.1322063

0:00:02.290008250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.2318082

0:00:02.390466250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.3322130

0:00:02.490474375 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.4322175

0:00:02.590485750 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.5322201

0:00:02.689727875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.6315118

0:00:02.789730125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.7315121

0:00:02.889733125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.8315113

0:00:02.989738250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.9315128

0:00:03.090405625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.0322022

0:00:03.190456250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.1322098

0:00:03.290431250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.2322115

0:00:03.390457500 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.3322173

0:00:03.490431375 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.4321940

0:00:03.590439625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.5321931

0:00:03.690431500 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.6321940

0:00:03.790451250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.7322051

0:00:03.890416750 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.8322015

0:00:03.990485625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.9322081

0:00:04.089677250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.0315103

0:00:04.190465750 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.1322150

0:00:04.290432875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.2322166

0:00:04.390434625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.3321926

0:00:04.490410125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.4321953

0:00:04.590455375 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.5322001

0:00:04.690407500 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.6322021

0:00:04.790412125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.7322062

^Chandling interrupt.

Interrupt: Stopping pipeline ...

Execution ended after 0:00:04.803064250

Setting pipeline to PAUSED ...

Sett[ 200.845472] ov5640_mipi 2-003c: s_stream: 0

ing pipeline to READY ...

Setting pipeline to NULL ...

Total showed frames (145), playing for (0:00:04.803314875), fps (30.187).

Freeing pipeline ...

------------

we didn't notice any frame loss

Regards

Re: v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

vincentz63 — Thu, 09 Jan 2020 14:46:49 GMT

You clearly are losing frames.

Read the log you posted: "lost frames detected: count = 1 - ts: 0:00:04.6322021"

Refer to my original gst-launch cmd line for a valid test.

Can we escalate this please? I've posted the question on 22nd Nov and you've only run a test now. The plugin is provided by your good selves to use the G2D graphics engine. It is documented in your own documentation. I'm very much ready to accept I'm doing something wrong, but at the rate at which things are progressing we'll never find out!