zero copy about v4l2 to drm in imx8qm

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

zero copy about v4l2 to drm in imx8qm

5,851 Views
yibhan
Contributor III

Hi guys,

As above subject, I have some trouble to implement zero copy about camera data to drm or display unit in i.MX8qm.

I think /unit_tests/V4L2/mx8_v4l2_cap_drm.out which is implemented by copying v4l2 camera data to drm it may cause delayed time to display camera data. So, I'd like to change  mx8_v4l2_cap_drm with zero copy for display camera data and i am using ov5640 mipi camera.

 

Do you have any ideas or sample codes or reference link?

 

0 Kudos
19 Replies

5,644 Views
RameshChandrakasan
Contributor III

Dear Yibhan

MY board : I.MX8QM - Freescale MEK

Linux Version:

repo init -u https://source.codeaurora.org/external/imx/imx-manifest -b imx-linux-sumo -m imx-4.14.98-2.3.0.xml

DTB: fsl-imx8qm-mek-hdmi.dtb

What i am doing :

/unit_tests/V4L2/mx8_v4l2_cap_drm.out -cam 1 -of
/unit_tests/V4L2/mx8_v4l2_cap_drm.out -cam 1
/unit_tests/V4L2/mx8_v4l2_cap_drm.out -cam 1 -m 0 -fr 30

Problem:

I am not getting any picture on Display

Pls help

0 Kudos

5,635 Views
prasad_imx8
Contributor III

Hi, I am prasad and i have some experience in the problem you are facing. My setup was slightly different.

however, for your observation the isuue might be in the video input or video display pipeline so you have to first see where is the problem. To identify if video input fails:
gst-launch-1.0 -vvv v4l2src device=/dev/video0 num-buffers=5 ! video/x-raw,width=1280,height=800 ! jpegenc ! filesink location=test_1280x800.jpg

To test display:
modetest -s <connector_id>[,<connector_id>][@<crtc_id>]:<mode>[-<vrefresh>][@<format>]

if the issue is in video input (more likely)

1) test setup is faulty.
2) Virtual channel is enabled in the CSI capture port and input stream doesnt have VC ID.
3) Number of data lanes mismatch.
4) Sync polarities.
5) Pinmuxing (I dont think this is the issue)
6) Color format mismatch (input sensor and CSI port)

I can help you more if you tell me, which camera sensor are you using? and the exact setup. 

Hope this helps.

Best Regards,
Prasad.

0 Kudos

5,617 Views
RameshChandrakasan
Contributor III

Dear Prasad

Thanks.

My camera is working when i use with gstreamer (ie) using gst-launch-1.0 command.

But , I want to get the picture without gstreamer framework. i have attached my log here, in that first command is using gstreamer,

But

Second command using raw driver is not working (ie)

$/unit_tests/V4L2/mx8_v4l2_cap_drm.out  --- which is my problem.

Pls suggest

Regards

Ramesh

 

0 Kudos

5,611 Views
prasad_imx8
Contributor III

Hey,

 

You have weston running in the background. The DRM can only have one master.

run this command before you run the unit test.

systemctl disable weston
killall weston

 

This will s

 

0 Kudos

3,868 Views
mallibeerala
Contributor III

Hi Prasad,

I have to run my camera with weston services enable, is there any way to do that, please suggest us.

Thanks & Regards,
Mallikarjuna B.

0 Kudos

5,607 Views
prasad_imx8
Contributor III

Hey,

You have weston running in the background. The DRM can only have one master.
run this command before you run the unit test.

systemctl disable weston
killall weston

This will solve the issue.

5,597 Views
RameshChandrakasan
Contributor III

Dear Prasad

Well Done. It is working as per the way you suggested.

Now, Next level.

Can we i use in side the wayland weston window context.? -Pls your suggestion

Because My graphics application is developed on wayland window.

As , "Zero copy is intended for this video display pipe line. it is matter of configuring the ISI , capture , display controller: -imx8  and finally connect and displaying them like digitally coupled streaming pipeline."

If i am using gstreamer frame work for this display application , really it is killing the i.mx8qm  hardware architecture. Processor is copying pixel by pixel and displaying on screen...i.mx8qm is platform (linux,widows,vxworks etc ) independent SOC. gstreamer is relally killing the i.mx8qm evaluation and on this direction. Making people to work on imaginary PADS,SINK,ELEMENT

Finally , Will it possible to do with raw driver on top of wayland? so that graphics and video can be overlayed/mixed?

Pls suggest.

Regards

Ramesh

0 Kudos

4,042 Views
marcus-castlepeakinc
Contributor III

I've got a 8MPLUSPD4-PEVK.  I tried to run

root@imx8mpevk:/unit_tests/V4L2# ./mx8_v4l2_cap_drm.out -cam 7 -fmt NV12 -of
(init_video_channel:513): init channel[0] save_file_name=0.NV12
(init_video_channel:517): init channel[0] v4l2_dev_name=/dev/video0 w/h=(1280,800)
(init_video_channel:513): init channel[1] save_file_name=1.NV12
(init_video_channel:517): init channel[1] v4l2_dev_name=/dev/video1 w/h=(1280,800)
(init_video_channel:513): init channel[2] save_file_name=2.NV12
(init_video_channel:517): init channel[2] v4l2_dev_name=/dev/video2 w/h=(1280,800)
(open_save_file:594): open 0.NV12 success
(open_save_file:594): open 1.NV12 success
(open_save_file:594): open 2.NV12 success
(main:1835): =*= success =*=

And got 3 zero length files.

Does anyone have any idea? 

Oh, and does anyone know where the source is to mx8_v4l2_cap_drm?

0 Kudos

5,592 Views
prasad_imx8
Contributor III

hello,

This is a very common issue we face with wayland. Yes there a way but I haven't tried it on NXP. You have to make some changes in the kernel so that there is no conflicting commit to the display. 

May be NXP should help you here.  

Best regards,
Prasad.

0 Kudos

5,566 Views
RameshChandrakasan
Contributor III

Dear Prasad,

I am trying to compile the mx6s_v4l2_cap_drm.c

/opt/fsl-imx-xwayland/4.14-sumo/sysroots/aarch64-poky-linux/usr/include/xf86drm.h:40:10: fatal error: drm.h: No such file or directory
#include <drm.h>
^~~~~~~
compilation terminated.
Makefile:34: recipe for target 'mx6s_v4l2_cap_drm.o' failed
make: *** [mx6s_v4l2_cap_drm.o] Error 1

Error log


Screenshot_drm.png

I have searched the file in target root file system.  Not found

Have you faced similar issues?

Regards

Ramesh

0 Kudos

5,550 Views
prasad_imx8
Contributor III

Locally include it from the linux kernel (copy paste the header file) 

And why are you compiling IMX6 (you are using IMX8 right?)

0 Kudos

5,684 Views
prasad_imx8
Contributor III

Hi, 

I was able to do it. may be we can discuss more.

Then post the final solution here.

Best Regards,

Prasad.

0 Kudos

5,784 Views
prasad_imx8
Contributor III

Hey! I am also trying to get a low latency sample application. Any progress on this yibhan?

Best regards,

prasad.

0 Kudos

5,773 Views
yibhan
Contributor III

Hi guys,

 

not yet

0 Kudos

5,845 Views
igorpadykov
NXP Employee
NXP Employee

Hi yibhan

 

one can try gstreamer camera encode example with zero-copy support :

gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 io-mode=dmabuf ! \
'video/x-raw,format=(string)NV12,width=1920,height=1080,framerate=(fraction)30/1' ! \
queue ! v4l2h264enc output-io-mode=dmabuf-import ! \
avimux ! filesink location=test.avi

 

Best regards
igor

0 Kudos

4,041 Views
marcus-castlepeakinc
Contributor III

I tried this on a 8MPLUSLPD4-PEVK and got:

21504.204670] enter isp_mi_stop
Setting pipeline to PAUSED ...

====== V4L2ENC: 1.18.5 build on Nov 9 2021 07:44:26. ======
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
[21504.558748] enter isp_mi_stop
[21505.215150] enter isp_s_comp
[21505.218062] enter isp_s_comp
[21505.220980] enter isp_s_comp
[21505.236736] enter wdr3_hw_init
[21505.239799] wdr3 res: 3840 2160
[21505.243742] enter isp_mi_start
ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: Failed to process frame.
Additional debug info:
../git/sys/v4l2/gstv4l2videoenc.c(833): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
Maybe be due to not enough memory or failing driver
Execution ended after 0:00:01.122470649
Setting pipeline to NULL ...
[21505.463975] enter isp_mi_stop
[21505.613430] enter isp_mi_stop
Freeing pipeline ...

 

Anyone have any ideas why this doesn't work?

0 Kudos

5,833 Views
yibhan
Contributor III

Hi igor,

Thank you for answer. But I'd like to set v4l2 and drm directly not using gst-lanuch.

Could you have any ideas or reference codes?

0 Kudos

5,258 Views
RameshChandrakasan
Contributor III

 

Dear Freescale -V4L2 Driver Experts

With Respect to the below  "kernel message"

Which  I.MX8 digital IP is represented by [mxc_isi.0]  & [mxc_isi.0.capture]?

(i.e) [ 5.548794] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]

mxc_isi.0 is receiving data from mxc-mipi-csi2.0 -- Understood , ISI is Receiving Video Data from MIPI CSI2 to Input port of ISI .

So, [mxc_isi.0] ----->Represent ISI Input Port

Similarly

[ 5.535194] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]

mxc_isi.0  supplying to mxc_isi.0.capture

Query : What is mxc_isi.0.capture? , Output of ISI , or what it represents?

---------------------------------------------Kernel Log Message -----------------------------------------

[ 5.477763] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[ 5.503068] mx8-img-md: Registered mxc_isi.4.capture as /dev/video1

[ 5.535194] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]

[ 5.548794] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]

[ 5.562380] mx8-img-md: created link [mxc_isi.4] => [mxc_isi.4.capture]

[ 5.575986] mx8-img-md: created link [mxc-mipi-csi2.1] => [mxc_isi.4]

[ 5.590799] mx8-img-md: created link [ov5640_mipi_v3 7-003c] => [mxc-mipi-csi2.0]

------------------------------------------Kernel Log Message Ends ------------------------------------

Pls Suggest

Regards

Ramesh

 

0 Kudos

5,244 Views
prasad_imx8
Contributor III

Hello,

 

I am not working for NXP nor an expert. But I know a little bit about this topic, so I will just tell you my opinion. NXP may differ.

 

The different naming comes from of course the things managed by the modules  and how NXP has segregated the configuration in the drivers.

mxc_isi.0: corresponds to driver imx8-isi-core.c, which is the driver for the ISI module. Handles clock, interrupt and stuff.

mxc_isi.0.capture: corresponds to driver  imx8-isi-cap.c. this handles buffer management and formats.

Rest of them you would already know.

The data flow can be Visualized as 

[ov5640_mipi_v3 7-003c] => [mxc-mipi-csi2.0]=> [mxc_isi.0] => [mxc_isi.0.capture]

For me mxc_isi.0 and mxc_isi.0.capture are single component just divided into to make it less complicated. 

Output of mxc_isi.0.capture is the capture video data.