v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

vincentz63 · ‎11-22-2019

I am experimenting with imxcompositor_g2d in order to overlay some text in an RTP pipeline but as soon as it is linked together with the v4l2src, the v4l2src element drops frames and prints "lost frames detected".

I'd like to use the hardware acceleration for the overlay as the textoverlay element consumes too much CPU and ditto for the compositor element.

I have boiled down the problem to the following very simple GStreamer pipeline:

imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! c.sink_0

This is running on an i.MX8M Mini.

The CPU usage is pretty low, around 7%. Profiling the app shows that the following functions are using up most of the CPU:

__pi___inval_dcache_area
__pi___clean_dcache_area_poc
_raw_spin_unlock_irqrestore

And an example of the callstack is below (taken from perf).

I'm guessing I'm doing something seriously wrong here as otherwise the imxcompositor_g2d would be unusable.

If I replace the imxcompositor_g2d with the compositor element, the CPU usage goes to 80%, but the v4l2 does not report dropped frames.

84.01%     0.00%  comp:src        libglib-2.0.so.0.5400.3           [.] g_thread_proxy
            |
            ---g_thread_proxy
               g_thread_pool_thread_proxy
               default_func
               gst_task_func
               |          
                --83.78%--gst_aggregator_aggregate_func
                          |          
                          |--81.03%--gst_video_aggregator_aggregate
                          |          |          
                          |          |--72.52%--gst_imxcompositor_aggregate_frames
                          |          |          |          
                          |          |          |--36.69%--imx_g2d_fill_color
                          |          |          |          |          
                          |          |          |          |--33.24%--__GI___ioctl
                          |          |          |          |          |          
                          |          |          |          |           --33.16%--__sys_trace_return
                          |          |          |          |                     sys_ioctl
                          |          |          |          |                     |          
                          |          |          |          |                      --33.10%--do_vfs_ioctl
                          |          |          |          |                                dma_buf_ioctl
                          |          |          |          |                                |          
                          |          |          |          |                                |--13.44%--dma_buf_map_attachment
                          |          |          |          |                                |          ion_map_dma_buf
                          |          |          |          |                                |          __pi___clean_dcache_area_poc
                          |          |          |          |                                |          
                          |          |          |          |                                |--13.00%--dma_buf_unmap_attachment
                          |          |          |          |                                |          ion_unmap_dma_buf
                          |          |          |          |                                |          |          
                          |          |          |          |                                |           --12.93%--__pi___inval_dcache_area
                          |          |          |          |                                |          
                          |          |          |          |                                 --6.11%--arch_setup_dma_ops
                          |          |          |          |                                           |          
                          |          |          |          |                                            --6.04%--of_find_compatible_node
                          |          |          |          |                                                      _raw_spin_unlock_irqrestore

Bio_TICFSL · ‎01-09-2020

Hello

We found a multimedia BSP with the specific plugin:

Environment:

i.MX 8MM EVK

Camera OV5640

BSP L4.19.35_1.1.0 multimedia image

here is the log:

-------------------

GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! waylandsink sync=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0

Setting pipeline to PAUSED ...

0:00:00.056040250 4072 0x3afa4c00 WARN v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4

L2_EVENT_EOS event for device '/dev/video0'.

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

[ 196.831569] alloc_contig_range: [81e00, 821f5) PFNs busy

[ 196.862006] ov5640_mipi 2-003c: s_stream: 1

[ 197.061215] skip frame 1

0:00:01.123702000 4072 0x3aef3370 WARN v4l2bufferpool gstv4l2bufferpool.c:1291:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2

_buffer.field to ANY

0:00:01.125192500 4072 0x3aef32d0 WARN videopool gstvideopool.c:228:video_buffer_pool_set_config:<waylandpool0> Provided size is to small for the cap

s: 0 < 4147200

0:00:01.132573750 4072 0x3aef32d0 FIXME basesink gstbasesink.c:3248:gst_base_sink_default_event:<waylandsink0> stream-start event without group-id. C

onsider implementing group-id handling in the upstream elements

0:00:01.139302125 4072 0x3aef32d0 WARN waylandsink wlwindow.c:333:gst_wl_window_new_internal: init surface_state fail, fallback to scale=1 fullscreen (

1920x1048)

0:00:01.756389875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 16 - ts: 0:00:01.698179

750

0:00:02.057064125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 7 - ts: 0:00:01.9988748

75

0:00:02.190468875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:02.1322063

75

0:00:02.290008250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.2318082

50

0:00:02.390466250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.3322130

00

0:00:02.490474375 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.4322175

00

0:00:02.590485750 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.5322201

25

0:00:02.689727875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.6315118

75

0:00:02.789730125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.7315121

25

0:00:02.889733125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.8315113

75

0:00:02.989738250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.9315128

75

0:00:03.090405625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.0322022

50

0:00:03.190456250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.1322098

75

0:00:03.290431250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.2322115

00

0:00:03.390457500 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.3322173

75

0:00:03.490431375 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.4321940

00

0:00:03.590439625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.5321931

25

0:00:03.690431500 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.6321940

00

0:00:03.790451250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.7322051

25

0:00:03.890416750 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.8322015

00

0:00:03.990485625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.9322081

25

0:00:04.089677250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.0315103

75

0:00:04.190465750 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.1322150

00

0:00:04.290432875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.2322166

25

0:00:04.390434625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.3321926

25

0:00:04.490410125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.4321953

75

0:00:04.590455375 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.5322001

25

0:00:04.690407500 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.6322021

25

0:00:04.790412125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.7322062

50

^Chandling interrupt.

Interrupt: Stopping pipeline ...

Execution ended after 0:00:04.803064250

Setting pipeline to PAUSED ...

Sett[ 200.845472] ov5640_mipi 2-003c: s_stream: 0

ing pipeline to READY ...

Setting pipeline to NULL ...

Total showed frames (145), playing for (0:00:04.803314875), fps (30.187).

Freeing pipeline ...

------------

we didn't notice any frame loss

Regards

vincentz63 · ‎01-09-2020

You clearly are losing frames.

Read the log you posted: "lost frames detected: count = 1 - ts: 0:00:04.6322021"

Refer to my original gst-launch cmd line for a valid test.

Can we escalate this please? I've posted the question on 22nd Nov and you've only run a test now. The plugin is provided by your good selves to use the G2D graphics engine. It is documented in your own documentation. I'm very much ready to accept I'm doing something wrong, but at the rate at which things are progressing we'll never find out!

Bio_TICFSL · ‎11-27-2019

Hello JP,

I am a little confused by your question. is you gstreamer gst-launch-1.0 command is

imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! c.sink_0?

This seems not a valid gstreamer command.

Could you provide your whole gstreamer command?

Regards

vincentz63 · ‎11-27-2019

Hello,

Sorry I have to manually type the command and I missed the caps filter on the V4L2SRC element.

GST_DEBUG=3 gst-launch-1.0 "imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0"

It prints the following warning: "v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected".

Bio_TICFSL · ‎12-06-2019

which BSP version are you using? I am using 4.14-sumo, I cannot run your command

ERROR: pipeline could not be constructed: no element "imxcompositor_g2d".

I tried both:

GST_DEBUG=3 gst-launch-1.0 "imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0"

and

gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0

Regards

vincentz63 · ‎12-09-2019

Hi,

You need to remove the quotes (")

So:

GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0

EDIT:

Forgot to sat that I'm using imx-yocto-L4.14.98_2.0.0_ga

Bio_TICFSL · ‎12-11-2019

Yes, I tried this one, not working. There is no imxcompositor_g2d in bsp I am using. Which bsp are your using?

And which board? imx8m mini board?

vincentz63 · ‎12-11-2019

It's for the i.MX8M Mini (as stated in the title of the question) and I am using the i.MX8MMini-EVK. The image is built for MACHINE=imx8mmevk and DISTRO=fsl-imx-xwayland. The image is fsl-image-validation-imx.

From memory, the element was present on the stock image present on the eMMC on the EVK.

The imxcompositor_g2d element is one of the elements provided by NXP.

It is somewhat documented in the "i.MX Linux User's Guide", page 50 "Video Composition": "imxcompositor_g2d uses corresponding hardware to accelerate video composition..."

vincentz63 · ‎12-12-2019

Hi,

Any chance of getting some feedback soon?

To be clear I'm using L4.14.98_2.0.0_ga on the i.MX8MMini-EVK.

Thanks

v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

i.MX 8M | i.MX 8M Mini | i.MX 8M Nano

Multimedia