v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

cancel
Showing results for 
Search instead for 
Did you mean: 

v4l2src lost frames detected when using imxcompositor_g2d on i.MX8MMini

452 Views
Contributor III

I am experimenting with imxcompositor_g2d in order to overlay some text in an RTP pipeline but as soon as it is linked together with the v4l2src, the v4l2src element drops frames and prints "lost frames detected".

I'd like to use the hardware acceleration for the overlay as the textoverlay element consumes too much CPU and ditto for the compositor element.

I have boiled down the problem to the following very simple GStreamer pipeline:

imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! c.sink_0

This is running on an i.MX8M Mini.

The CPU usage is pretty low, around 7%. Profiling the app shows that the following functions are using up most of the CPU:

  • __pi___inval_dcache_area
  • __pi___clean_dcache_area_poc
  • _raw_spin_unlock_irqrestore

And an example of the callstack is below (taken from perf).

I'm guessing I'm doing something seriously wrong here as otherwise the imxcompositor_g2d would be unusable.

If I replace the imxcompositor_g2d with the compositor element, the CPU usage goes to 80%, but the v4l2 does not report dropped frames.

84.01%     0.00%  comp:src        libglib-2.0.so.0.5400.3           [.] g_thread_proxy
            |
            ---g_thread_proxy
               g_thread_pool_thread_proxy
               default_func
               gst_task_func
               |          
                --83.78%--gst_aggregator_aggregate_func
                          |          
                          |--81.03%--gst_video_aggregator_aggregate
                          |          |          
                          |          |--72.52%--gst_imxcompositor_aggregate_frames
                          |          |          |          
                          |          |          |--36.69%--imx_g2d_fill_color
                          |          |          |          |          
                          |          |          |          |--33.24%--__GI___ioctl
                          |          |          |          |          |          
                          |          |          |          |           --33.16%--__sys_trace_return
                          |          |          |          |                     sys_ioctl
                          |          |          |          |                     |          
                          |          |          |          |                      --33.10%--do_vfs_ioctl
                          |          |          |          |                                dma_buf_ioctl
                          |          |          |          |                                |          
                          |          |          |          |                                |--13.44%--dma_buf_map_attachment
                          |          |          |          |                                |          ion_map_dma_buf
                          |          |          |          |                                |          __pi___clean_dcache_area_poc
                          |          |          |          |                                |          
                          |          |          |          |                                |--13.00%--dma_buf_unmap_attachment
                          |          |          |          |                                |          ion_unmap_dma_buf
                          |          |          |          |                                |          |          
                          |          |          |          |                                |           --12.93%--__pi___inval_dcache_area
                          |          |          |          |                                |          
                          |          |          |          |                                 --6.11%--arch_setup_dma_ops
                          |          |          |          |                                           |          
                          |          |          |          |                                            --6.04%--of_find_compatible_node
                          |          |          |          |                                                      _raw_spin_unlock_irqrestore
0 Kudos
9 Replies

66 Views
NXP TechSupport
NXP TechSupport

Hello

We found a multimedia BSP with the specific plugin:

 

Environment:

i.MX 8MM EVK

Camera OV5640

BSP L4.19.35_1.1.0 multimedia image

 

here is the log:

 

-------------------

GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! waylandsink sync=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0

 

Setting pipeline to PAUSED ...

0:00:00.056040250  4072     0x3afa4c00 WARN                    v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4

L2_EVENT_EOS event for device '/dev/video0'.

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

[  196.831569] alloc_contig_range: [81e00, 821f5) PFNs busy

[  196.862006] ov5640_mipi 2-003c: s_stream: 1

[  197.061215] skip frame 1

0:00:01.123702000  4072     0x3aef3370 WARN          v4l2bufferpool gstv4l2bufferpool.c:1291:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2

_buffer.field to ANY

0:00:01.125192500  4072     0x3aef32d0 WARN               videopool gstvideopool.c:228:video_buffer_pool_set_config:<waylandpool0> Provided size is to small for the cap

s: 0 < 4147200

0:00:01.132573750  4072     0x3aef32d0 FIXME               basesink gstbasesink.c:3248:gst_base_sink_default_event:<waylandsink0> stream-start event without group-id. C

onsider implementing group-id handling in the upstream elements

0:00:01.139302125  4072     0x3aef32d0 WARN             waylandsink wlwindow.c:333:gst_wl_window_new_internal: init surface_state fail, fallback to scale=1 fullscreen (

1920x1048)

0:00:01.756389875  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 16 - ts: 0:00:01.698179

750

0:00:02.057064125  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 7 - ts: 0:00:01.9988748

75

0:00:02.190468875  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:02.1322063

75

0:00:02.290008250  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.2318082

50

0:00:02.390466250  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.3322130

00

0:00:02.490474375  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.4322175

00

0:00:02.590485750  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.5322201

25

0:00:02.689727875  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.6315118

75

0:00:02.789730125  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.7315121

25

0:00:02.889733125  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.8315113

75

0:00:02.989738250  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.9315128

75

0:00:03.090405625  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.0322022

50

0:00:03.190456250  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.1322098

75

0:00:03.290431250  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.2322115

00

0:00:03.390457500  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.3322173

75

0:00:03.490431375  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.4321940

00

0:00:03.590439625  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.5321931

25

0:00:03.690431500  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.6321940

00

0:00:03.790451250  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.7322051

25

0:00:03.890416750  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.8322015

00

0:00:03.990485625  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.9322081

25

0:00:04.089677250  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.0315103

75

0:00:04.190465750  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.1322150

00

0:00:04.290432875  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.2322166

25

0:00:04.390434625  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.3321926

25

0:00:04.490410125  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.4321953

75

0:00:04.590455375  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.5322001

25

0:00:04.690407500  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.6322021

25

0:00:04.790412125  4072     0x3aef3370 WARN                 v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.7322062

50

^Chandling interrupt.

Interrupt: Stopping pipeline ...

Execution ended after 0:00:04.803064250

Setting pipeline to PAUSED ...

Sett[  200.845472] ov5640_mipi 2-003c: s_stream: 0

ing pipeline to READY ...

Setting pipeline to NULL ...

Total showed frames (145), playing for (0:00:04.803314875), fps (30.187).

Freeing pipeline ...

------------

 

we didn't notice any frame loss

Regards

0 Kudos

66 Views
Contributor III

You clearly are losing frames.

Read the log you posted: "lost frames detected: count = 1 - ts: 0:00:04.6322021"

Refer to my original gst-launch cmd line for a valid test.

Can we escalate this please? I've posted the question on 22nd Nov and you've only run a test now. The plugin is provided by your good selves to use the G2D graphics engine. It is documented in your own documentation. I'm very much ready to accept I'm doing something wrong, but at the rate at which things are progressing we'll never find out!

0 Kudos

66 Views
NXP TechSupport
NXP TechSupport

Hello JP,

I am a little confused by your question.  is you gstreamer gst-launch-1.0  command is

imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! c.sink_0?

 

This seems not a valid gstreamer command.

 

Could you provide your whole gstreamer command?  

Regards

0 Kudos

66 Views
Contributor III

Hello,

Sorry I have to manually type the command and I missed the caps filter on the V4L2SRC element.

GST_DEBUG=3 gst-launch-1.0 "imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0"

It prints the following warning: "v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected".

0 Kudos

66 Views
NXP TechSupport
NXP TechSupport

which BSP version are you using?  I am using 4.14-sumo,  I cannot run your command

ERROR: pipeline could not be constructed: no element "imxcompositor_g2d".

 

I tried both:

GST_DEBUG=3 gst-launch-1.0 "imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0"

 

and

gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0

Regards

0 Kudos

66 Views
Contributor III

Hi,

You need to remove the quotes (")

So:

GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0

EDIT:

Forgot to sat that I'm using imx-yocto-L4.14.98_2.0.0_ga

0 Kudos

66 Views
NXP TechSupport
NXP TechSupport

Yes, I tried this one, not working.  There is no imxcompositor_g2d  in bsp I am using.   Which bsp are your using?

And which board?  imx8m mini board?

0 Kudos

66 Views
Contributor III

It's for the i.MX8M Mini (as stated in the title of the question) and I am using the i.MX8MMini-EVK. The image is built for MACHINE=imx8mmevk and DISTRO=fsl-imx-xwayland. The image is fsl-image-validation-imx.

From memory, the element was present on the stock image present on the eMMC on the EVK.

The imxcompositor_g2d element is one of the elements provided by NXP.

It is somewhat documented in the "i.MX Linux User's Guide", page 50 "Video Composition": "imxcompositor_g2d uses corresponding hardware to accelerate video composition..."

0 Kudos

66 Views
Contributor III

Hi,

Any chance of getting some feedback soon?

To be clear I'm using L4.14.98_2.0.0_ga on the i.MX8MMini-EVK.

Thanks

0 Kudos