AnsweredAssumed Answered

Capturing YUV420 data over mipi csi

Question asked by Efe Can İçöz on Apr 30, 2020
Latest reply on May 1, 2020 by igorpadykov

Hi all,

 

I'm trying to capture camera data with strict latency limit to ~100ms. Currently i've connected a camera to my mipi csi bus (4 lane 786mhz) and streaming over standard yuv422/uyvy format. Since wayland doesn't support UYVY (gst-inspect says that it is supporting but in source UYVY doesn't supported in DMAbuf) as input i need to use imxvideoconvert_g2d and with that configuration i'm having ~160ms latency, command is below.

"gst-launch-1.0 v4l2src device="/dev/video0" ! "video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1080, framerate=(fraction)25/1" ! imxvideoconvert_g2d !  waylandsink"

 

I did not want to dive into wayland side to make it possible to stream uyvy, i want to switch yuv420 which i assume has the format NV12 and which i see every device (wayland + mipi port of imx8mm + my cam) supports it natively. Even though i found out that mx6s_capture driver (driver of mipi csi) doesn't cover the yuv420 implementation even the hw supports it.

 

I'll try to add the support of yuv420 to that driver but my initial trials have failed, is there any guide or anyone that has already done that before it would be nice to hear the details. Otherwise it's being quite hard to learn&understand the mipi driver.

 

Have a nice day,

Efe

 

 

Edit 1:

Currently i'm capturing image like below with this command:

`v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=NV12 --stream-mmap --stream-count=1 --stream-to=grab-nv12.raw`

 

And gstreamer with command below gives me this error:

v4l2src device="/dev/video0" ! "video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, framerate=(fraction)25/1" !  waylandsink

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstWaylandSink:waylandsink0: Video memory can not be read from userspace.
Additional debug info:
../../../git/ext/wayland/gstwaylandsink.c(1127): gst_wayland_sink_show_frame (): /GstPipeline:pipeline0/GstWaylandSink:waylandsink0
Execution ended after 0:00:00.451851860
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

With the debug level 5 erroneous point is here:

0:00:00.862928777 2444 0x2e28ef70 DEBUG waylandsink wllinuxdmabuf.c:108:gst_wl_linux_dmabuf_construct_wl_buffer:<GstWlDisplay@0x2e36b810> Creating wl_buffer from DMABuf of size 3133440 (1920 x 1080), format NV12 modifier 0x0000000000000000
0:00:00.863007404 2444 0x2e28ef70 DEBUG GST_META gstmeta.c:88:gst_meta_api_type_register: register API "GstPhyMemMetaAPI"
0:00:00.863058654 2444 0x2e28ef70 DEBUG GST_META gstmeta.c:95:gst_meta_api_type_register: adding tag "video"
0:00:00.863079280 2444 0x2e28ef70 DEBUG GST_META gstmeta.c:95:gst_meta_api_type_register: adding tag "size"
0:00:00.863097655 2444 0x2e28ef70 DEBUG GST_META gstmeta.c:95:gst_meta_api_type_register: adding tag "orientation"
0:00:00.864253798 2444 0x2e28ef70 DEBUG waylandsink wllinuxdmabuf.c:175:gst_wl_linux_dmabuf_construct_wl_buffer:<ionallocator0> created linux_dmabuf wl_buffer (0xffff80003c80):1920x1080, fmt=NV12, 2 planes
0:00:00.864344675 2444 0x2e28ef70 DEBUG GST_PERFORMANCE gstminiobject.c:524:ensure_priv_data: allocating private data GstBuffer miniobject 0x2e36db40
0:00:00.864274673 2444 0x2e28ee80 DEBUG GST_POLL gstpoll.c:1414:gst_poll_wait: 0x2e28ee30: timeout :99:99:99.999999999
0:00:00.864486427 2444 0x2e28ef70 DEBUG fdmemory gstfdmemory.c:136:gst_fd_mem_map: 0xffff78024420: fd 29: mapped 0xffff85e9f000
0:00:00.864533678 2444 0x2e28ef70 DEBUG fdmemory gstfdmemory.c:136:gst_fd_mem_map: 0x2e1d46e0: fd 18: mapped 0xffff85ba7000
0:00:00.864605804 2444 0x2e28ef70 ERROR videometa gstvideometa.c:240:default_map: plane 1, no memory at offset 3110400
0:00:00.864628054 2444 0x2e28ef70 ERROR default video-frame.c:162:gst_video_frame_map_id: failed to map video frame plane 1
0:00:00.864666055 2444 0x2e28ef70 WARN waylandsink gstwaylandsink.c:1127:gst_wayland_sink_show_frame:<waylandsink0> error: Video memory can not be read from userspace.
0:00:00.864693180 2444 0x2e28ef70 DEBUG GST_MESSAGE gstelement.c:2110:gst_element_message_full_with_details:<waylandsink0> start
0:00:00.864761681 2444 0x2e28ef70 INFO GST_ERROR_SYSTEM gstelement.c:2141:gst_element_message_full_with_details:<waylandsink0> posting message: Video memory can not be read from userspace.

Outcomes