zero copy about v4l2 to drm in imx8qm

[yibhan]

Hi guys,

As above subject, I have some trouble to implement zero copy about camera data to drm or display unit in i.MX8qm.

I think /unit_tests/V4L2/mx8_v4l2_cap_drm.out which is implemented by copying v4l2 camera data to drm it may cause delayed time to display camera data. So, I'd like to change  mx8_v4l2_cap_drm with zero copy for display camera data and i am using ov5640 mipi camera.

Do you have any ideas or sample codes or reference link?

[RameshChandrakasan]

Dear Yibhan

MY board : I.MX8QM - Freescale MEK

Linux Version:

repo init -u https://source.codeaurora.org/external/imx/imx-manifest -b imx-linux-sumo -m imx-4.14.98-2.3.0.xml

DTB: fsl-imx8qm-mek-hdmi.dtb

What i am doing :

/unit_tests/V4L2/mx8_v4l2_cap_drm.out -cam 1 -of
/unit_tests/V4L2/mx8_v4l2_cap_drm.out -cam 1
/unit_tests/V4L2/mx8_v4l2_cap_drm.out -cam 1 -m 0 -fr 30

Problem:

I am not getting any picture on Display

Pls help

[prasad_imx8]

Hi, I am prasad and i have some experience in the problem you are facing. My setup was slightly different.

however, for your observation the isuue might be in the video input or video display pipeline so you have to first see where is the problem. To identify if video input fails:
gst-launch-1.0 -vvv v4l2src device=/dev/video0 num-buffers=5 ! video/x-raw,width=1280,height=800 ! jpegenc ! filesink location=test_1280x800.jpg

To test display:
modetest -s <connector_id>[,<connector_id>][@<crtc_id>]:<mode>[-<vrefresh>][@<format>]

if the issue is in video input (more likely)

1) test setup is faulty.
2) Virtual channel is enabled in the CSI capture port and input stream doesnt have VC ID.
3) Number of data lanes mismatch.
4) Sync polarities.
5) Pinmuxing (I dont think this is the issue)
6) Color format mismatch (input sensor and CSI port)

I can help you more if you tell me, which camera sensor are you using? and the exact setup. 

Hope this helps.

Best Regards,
Prasad.

[RameshChandrakasan]

Dear Prasad

Thanks.

My camera is working when i use with gstreamer (ie) using gst-launch-1.0 command.

But , I want to get the picture without gstreamer framework. i have attached my log here, in that first command is using gstreamer,

But

Second command using raw driver is not working (ie)

$/unit_tests/V4L2/mx8_v4l2_cap_drm.out  --- which is my problem.

Pls suggest

Regards

Ramesh

[prasad_imx8]

Hey,

You have weston running in the background. The DRM can only have one master.

run this command before you run the unit test.

systemctl disable weston
killall weston

This will s

[mallibeerala]

Hi Prasad,

I have to run my camera with weston services enable, is there any way to do that, please suggest us.

Thanks & Regards,
Mallikarjuna B.

[prasad_imx8]

Hey,

You have weston running in the background. The DRM can only have one master.
run this command before you run the unit test.

systemctl disable weston
killall weston

This will solve the issue.

[RameshChandrakasan]

Dear Prasad

Well Done. It is working as per the way you suggested.

Now, Next level.

Can we i use in side the wayland weston window context.? -Pls your suggestion

Because My graphics application is developed on wayland window.

As , "Zero copy is intended for this video display pipe line. it is matter of configuring the ISI , capture , display controller: -imx8  and finally connect and displaying them like digitally coupled streaming pipeline."

If i am using gstreamer frame work for this display application , really it is killing the i.mx8qm  hardware architecture. Processor is copying pixel by pixel and displaying on screen...i.mx8qm is platform (linux,widows,vxworks etc ) independent SOC. gstreamer is relally killing the i.mx8qm evaluation and on this direction. Making people to work on imaginary PADS,SINK,ELEMENT

Finally , Will it possible to do with raw driver on top of wayland? so that graphics and video can be overlayed/mixed?

Pls suggest.

Regards

Ramesh

[marcus-castlepeakinc]

I've got a 8MPLUSPD4-PEVK.  I tried to run

root@imx8mpevk:/unit_tests/V4L2# ./mx8_v4l2_cap_drm.out -cam 7 -fmt NV12 -of
(init_video_channel:513): init channel[0] save_file_name=0.NV12
(init_video_channel:517): init channel[0] v4l2_dev_name=/dev/video0 w/h=(1280,800)
(init_video_channel:513): init channel[1] save_file_name=1.NV12
(init_video_channel:517): init channel[1] v4l2_dev_name=/dev/video1 w/h=(1280,800)
(init_video_channel:513): init channel[2] save_file_name=2.NV12
(init_video_channel:517): init channel[2] v4l2_dev_name=/dev/video2 w/h=(1280,800)
(open_save_file:594): open 0.NV12 success
(open_save_file:594): open 1.NV12 success
(open_save_file:594): open 2.NV12 success
(main:1835): =*= success =*=

And got 3 zero length files.

Does anyone have any idea? 

Oh, and does anyone know where the source is to mx8_v4l2_cap_drm?

[prasad_imx8]

hello,

This is a very common issue we face with wayland. Yes there a way but I haven't tried it on NXP. You have to make some changes in the kernel so that there is no conflicting commit to the display. 

May be NXP should help you here.  

Best regards,
Prasad.

[RameshChandrakasan]

Dear Prasad,

I am trying to compile the mx6s_v4l2_cap_drm.c

/opt/fsl-imx-xwayland/4.14-sumo/sysroots/aarch64-poky-linux/usr/include/xf86drm.h:40:10: fatal error: drm.h: No such file or directory
#include <drm.h>
^~~~~~~
compilation terminated.
Makefile:34: recipe for target 'mx6s_v4l2_cap_drm.o' failed
make: *** [mx6s_v4l2_cap_drm.o] Error 1

Error log

I have searched the file in target root file system.  Not found

Have you faced similar issues?

Regards

Ramesh

[prasad_imx8]

Locally include it from the linux kernel (copy paste the header file) 

And why are you compiling IMX6 (you are using IMX8 right?)

[prasad_imx8]

Hi, 

I was able to do it. may be we can discuss more.

Then post the final solution here.

Best Regards,

Prasad.

[prasad_imx8]

Hey! I am also trying to get a low latency sample application. Any progress on this yibhan?

Best regards,

prasad.

[yibhan]

Hi guys,

not yet

[igorpadykov]

Hi yibhan

one can try gstreamer camera encode example with zero-copy support :

gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 io-mode=dmabuf ! \
'video/x-raw,format=(string)NV12,width=1920,height=1080,framerate=(fraction)30/1' ! \
queue ! v4l2h264enc output-io-mode=dmabuf-import ! \
avimux ! filesink location=test.avi

Best regards
igor

[marcus-castlepeakinc]

I tried this on a 8MPLUSLPD4-PEVK and got:

21504.204670] enter isp_mi_stop
Setting pipeline to PAUSED ...

====== V4L2ENC: 1.18.5 build on Nov 9 2021 07:44:26. ======
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
[21504.558748] enter isp_mi_stop
[21505.215150] enter isp_s_comp
[21505.218062] enter isp_s_comp
[21505.220980] enter isp_s_comp
[21505.236736] enter wdr3_hw_init
[21505.239799] wdr3 res: 3840 2160
[21505.243742] enter isp_mi_start
ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: Failed to process frame.
Additional debug info:
../git/sys/v4l2/gstv4l2videoenc.c(833): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
Maybe be due to not enough memory or failing driver
Execution ended after 0:00:01.122470649
Setting pipeline to NULL ...
[21505.463975] enter isp_mi_stop
[21505.613430] enter isp_mi_stop
Freeing pipeline ...

Anyone have any ideas why this doesn't work?

[yibhan]

Hi igor,

Thank you for answer. But I'd like to set v4l2 and drm directly not using gst-lanuch.

Could you have any ideas or reference codes?

[RameshChandrakasan]

Dear Freescale -V4L2 Driver Experts

With Respect to the below  "kernel message"

Which  I.MX8 digital IP is represented by [mxc_isi.0]  & [mxc_isi.0.capture]?

(i.e) [ 5.548794] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]

mxc_isi.0 is receiving data from mxc-mipi-csi2.0 -- Understood , ISI is Receiving Video Data from MIPI CSI2 to Input port of ISI .

So, [mxc_isi.0] ----->Represent ISI Input Port

Similarly

[ 5.535194] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]

mxc_isi.0  supplying to mxc_isi.0.capture

Query : What is mxc_isi.0.capture? , Output of ISI , or what it represents?

---------------------------------------------Kernel Log Message -----------------------------------------

[ 5.477763] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[ 5.503068] mx8-img-md: Registered mxc_isi.4.capture as /dev/video1

[ 5.535194] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]

[ 5.548794] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]

[ 5.562380] mx8-img-md: created link [mxc_isi.4] => [mxc_isi.4.capture]

[ 5.575986] mx8-img-md: created link [mxc-mipi-csi2.1] => [mxc_isi.4]

[ 5.590799] mx8-img-md: created link [ov5640_mipi_v3 7-003c] => [mxc-mipi-csi2.0]

------------------------------------------Kernel Log Message Ends ------------------------------------

Pls Suggest

Regards

Ramesh

[prasad_imx8]

Hello,

I am not working for NXP nor an expert. But I know a little bit about this topic, so I will just tell you my opinion. NXP may differ.

The different naming comes from of course the things managed by the modules  and how NXP has segregated the configuration in the drivers.

mxc_isi.0: corresponds to driver imx8-isi-core.c, which is the driver for the ISI module. Handles clock, interrupt and stuff.

mxc_isi.0.capture: corresponds to driver  imx8-isi-cap.c. this handles buffer management and formats.

Rest of them you would already know.

The data flow can be Visualized as 

[ov5640_mipi_v3 7-003c] => [mxc-mipi-csi2.0]=> [mxc_isi.0] => [mxc_isi.0.capture]

For me mxc_isi.0 and mxc_isi.0.capture are single component just divided into to make it less complicated. 

Output of mxc_isi.0.capture is the capture video data.