I am experimenting with imxcompositor_g2d in order to overlay some text in an RTP pipeline but as soon as it is linked together with the v4l2src, the v4l2src element drops frames and prints "lost frames detected".
I'd like to use the hardware acceleration for the overlay as the textoverlay element consumes too much CPU and ditto for the compositor element.
I have boiled down the problem to the following very simple GStreamer pipeline:
imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! c.sink_0
This is running on an i.MX8M Mini.
The CPU usage is pretty low, around 7%. Profiling the app shows that the following functions are using up most of the CPU:
And an example of the callstack is below (taken from perf).
I'm guessing I'm doing something seriously wrong here as otherwise the imxcompositor_g2d would be unusable.
If I replace the imxcompositor_g2d with the compositor element, the CPU usage goes to 80%, but the v4l2 does not report dropped frames.
84.01% 0.00% comp:src libglib-2.0.so.0.5400.3 [.] g_thread_proxy | ---g_thread_proxy g_thread_pool_thread_proxy default_func gst_task_func | --83.78%--gst_aggregator_aggregate_func | |--81.03%--gst_video_aggregator_aggregate | | | |--72.52%--gst_imxcompositor_aggregate_frames | | | | | |--36.69%--imx_g2d_fill_color | | | | | | | |--33.24%--__GI___ioctl | | | | | | | | | --33.16%--__sys_trace_return | | | | sys_ioctl | | | | | | | | | --33.10%--do_vfs_ioctl | | | | dma_buf_ioctl | | | | | | | | | |--13.44%--dma_buf_map_attachment | | | | | ion_map_dma_buf | | | | | __pi___clean_dcache_area_poc | | | | | | | | | |--13.00%--dma_buf_unmap_attachment | | | | | ion_unmap_dma_buf | | | | | | | | | | | --12.93%--__pi___inval_dcache_area | | | | | | | | | --6.11%--arch_setup_dma_ops | | | | | | | | | --6.04%--of_find_compatible_node | | | | _raw_spin_unlock_irqrestore
Hello
We found a multimedia BSP with the specific plugin:
Environment:
i.MX 8MM EVK
Camera OV5640
BSP L4.19.35_1.1.0 multimedia image
here is the log:
-------------------
GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! waylandsink sync=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0
Setting pipeline to PAUSED ...
0:00:00.056040250 4072 0x3afa4c00 WARN v4l2 v4l2_calls.c:548:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4
L2_EVENT_EOS event for device '/dev/video0'.
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 196.831569] alloc_contig_range: [81e00, 821f5) PFNs busy
[ 196.862006] ov5640_mipi 2-003c: s_stream: 1
[ 197.061215] skip frame 1
0:00:01.123702000 4072 0x3aef3370 WARN v4l2bufferpool gstv4l2bufferpool.c:1291:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool:src> Driver should never set v4l2
_buffer.field to ANY
0:00:01.125192500 4072 0x3aef32d0 WARN videopool gstvideopool.c:228:video_buffer_pool_set_config:<waylandpool0> Provided size is to small for the cap
s: 0 < 4147200
0:00:01.132573750 4072 0x3aef32d0 FIXME basesink gstbasesink.c:3248:gst_base_sink_default_event:<waylandsink0> stream-start event without group-id. C
onsider implementing group-id handling in the upstream elements
0:00:01.139302125 4072 0x3aef32d0 WARN waylandsink wlwindow.c:333:gst_wl_window_new_internal: init surface_state fail, fallback to scale=1 fullscreen (
1920x1048)
0:00:01.756389875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 16 - ts: 0:00:01.698179
750
0:00:02.057064125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 7 - ts: 0:00:01.9988748
75
0:00:02.190468875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:02.1322063
75
0:00:02.290008250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.2318082
50
0:00:02.390466250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.3322130
00
0:00:02.490474375 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.4322175
00
0:00:02.590485750 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.5322201
25
0:00:02.689727875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.6315118
75
0:00:02.789730125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.7315121
25
0:00:02.889733125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.8315113
75
0:00:02.989738250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.9315128
75
0:00:03.090405625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.0322022
50
0:00:03.190456250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.1322098
75
0:00:03.290431250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.2322115
00
0:00:03.390457500 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.3322173
75
0:00:03.490431375 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.4321940
00
0:00:03.590439625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.5321931
25
0:00:03.690431500 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.6321940
00
0:00:03.790451250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.7322051
25
0:00:03.890416750 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.8322015
00
0:00:03.990485625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.9322081
25
0:00:04.089677250 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.0315103
75
0:00:04.190465750 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.1322150
00
0:00:04.290432875 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.2322166
25
0:00:04.390434625 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.3321926
25
0:00:04.490410125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.4321953
75
0:00:04.590455375 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.5322001
25
0:00:04.690407500 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.6322021
25
0:00:04.790412125 4072 0x3aef3370 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.7322062
50
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:04.803064250
Setting pipeline to PAUSED ...
Sett[ 200.845472] ov5640_mipi 2-003c: s_stream: 0
ing pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (145), playing for (0:00:04.803314875), fps (30.187).
Freeing pipeline ...
------------
we didn't notice any frame loss
Regards
You clearly are losing frames.
Read the log you posted: "lost frames detected: count = 1 - ts: 0:00:04.6322021"
Refer to my original gst-launch cmd line for a valid test.
Can we escalate this please? I've posted the question on 22nd Nov and you've only run a test now. The plugin is provided by your good selves to use the G2D graphics engine. It is documented in your own documentation. I'm very much ready to accept I'm doing something wrong, but at the rate at which things are progressing we'll never find out!
Hello JP,
I am a little confused by your question. is you gstreamer gst-launch-1.0 command is
imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! c.sink_0?
This seems not a valid gstreamer command.
Could you provide your whole gstreamer command?
Regards
Hello,
Sorry I have to manually type the command and I missed the caps filter on the V4L2SRC element.
GST_DEBUG=3 gst-launch-1.0 "imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0"
It prints the following warning: "v4l2src gstv4l2src.c:976:gst_v4l2src_create:<v4l2src0> lost frames detected".
which BSP version are you using? I am using 4.14-sumo, I cannot run your command
ERROR: pipeline could not be constructed: no element "imxcompositor_g2d".
I tried both:
GST_DEBUG=3 gst-launch-1.0 "imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0"
and
gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0
Regards
Hi,
You need to remove the quotes (")
So:
GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c ! video/x-raw,format=RGB16,width=1920,height=1080 ! fakevideosink sync=false async=true enable-last-sample=false v4l2src ! video/x-raw,format=YUY2,width=1920,height=1080 ! c.sink_0
EDIT:
Forgot to sat that I'm using imx-yocto-L4.14.98_2.0.0_ga
Yes, I tried this one, not working. There is no imxcompositor_g2d in bsp I am using. Which bsp are your using?
And which board? imx8m mini board?
It's for the i.MX8M Mini (as stated in the title of the question) and I am using the i.MX8MMini-EVK. The image is built for MACHINE=imx8mmevk and DISTRO=fsl-imx-xwayland. The image is fsl-image-validation-imx.
From memory, the element was present on the stock image present on the eMMC on the EVK.
The imxcompositor_g2d element is one of the elements provided by NXP.
It is somewhat documented in the "i.MX Linux User's Guide", page 50 "Video Composition": "imxcompositor_g2d uses corresponding hardware to accelerate video composition..."
Hi,
Any chance of getting some feedback soon?
To be clear I'm using L4.14.98_2.0.0_ga on the i.MX8MMini-EVK.
Thanks