iMX6DQ ISL79985/79987 MIPI CSI2 CVBS camera surround view solution for Linux BSP

[#1]

For iMX6DQ, there are two IPUs, so they can support up to 4 cameras at the same time. But the default BSP can only support up to two cameras at the same time.

The attached patch can make the BSP support up to 4 cameras based on 3.10.53 GA 1.1.0 BSP.

The 4 cameras can be:

- 1xCSI, 3xMIPI

- 2xCSI, 2xMIPI

- 4xMIPI

For 4xMIPI case, the four cameras should be combined on the single MIPI CSI2 interface, and each camera data should be transfered on a mipi virtual channel.

In this patch, we given the example driver for Intersil ISL79985. The input to ISL79985 is four CVBS camera.

There are four patches:

0001-IPU-update-IPU-capture-driver-to-support-up-to-four-.patch

     Updated IPU common code to support up to four cameras.

0002-Add-Intersil-ISL79985-MIPI-Video-Decoder-Driver-for-.patch

     ISL79985 driver, which can support both 1 lanes and 2 lanes mode.

0003-Remove-the-page-size-align-requirement-for-v4l2-capt.patch

     With this patch, the mxc_v4l2_tvin test application can use overlay framebuffer as V4l2 capture buffer directly.

0004-IPU-CSI-Drop-1-2-frame-on-MIPI-interface-for-interla.patch

     This patch is option, it will drop one field data, so for each camera, the input will be 720*240 30 FPS.

For 720P HD solution, it is based on Maxim MAX9286: iMX6DQ MAX9286 MIPI CSI2 720P camera surround view solution for Linux BSP

How to builld the kernel with ISL79985 support:

      make imx_v7_defconfig

      make menuconfig (In this command, you should select the ISL79985 driver:

            Device Drivers  --->

                  <*> Multimedia support  --->

                        [*]   V4L platform devices  --->

                              <*>   MXC Video For Linux Video Capture

                                      MXC Camera/V4L2 PRP Features support  --->

                                          <*>Intersil ISL79985 Video Decoder support

                                          <*>mxc VADC support

                                          <*>Select Overlay Rounting (Queue ipu device for overlay library)

                                          <*>Pre-processor Encoder library

                                          <*>IPU CSI Encoder library)

      make zImage

      make dtbs

The built out image file:

      arch/arm/boot/dts/imx6q-sabresd.dtb

      arch/arm/boot/zImage

"mxc_v4l2_tvin.zip" is the test application, test command to capture the four cameras and render on 1080P HDMI display:

/mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 1 -x 0 -g2d &

/mxc_v4l2_tvin.out -ol 960 -ot 0 -ow 960 -oh 540 -d 1 -x 1 -g2d &

/mxc_v4l2_tvin.out -ol 0 -ot 540 -ow 960 -oh 540 -d 1 -x 2 -g2d &
/mxc_v4l2_tvin.out -ol 960 -ot 540 -ow 960 -oh 540 -d 1 -x 3 -g2d &

2015-10-10 Update:

Updated the test application "mxc_v4l2_tvin_isl79985.tar.gz" to fix the Yocto build errors.

Updated ISL79985 register setting "page5, isl79985_write_reg(0x07, 0x46)" in patch "0002-Add-Intersil-ISL79985-MIPI-Video-Decoder-Driver-for-.patch", which can fix the green line issue.

2016-01-25 Update:

Added de-interlace support, L3.10.53_ISL79985_Surroundview_Patch_20160125.tar.gz

New test capplication for de-interlance: mxc_v4l2_tvin_isl79985_vdi_20160125.tar.gz

New test commands:

/mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 1 -x 0 -g2d -m &

/mxc_v4l2_tvin.out -ol 960 -ot 0 -ow 960 -oh 540 -d 1 -x 1 -g2d -m &

/mxc_v4l2_tvin.out -ol 0 -ot 540 -ow 960 -oh 540 -d 1 -x 2 -g2d -m &
/mxc_v4l2_tvin.out -ol 960 -ot 540 -ow 960 -oh 540 -d 1 -x 3 -g2d -m &

Note:  with the 0005-Add-interlaced-mode-capture-for-ISL79985.patch, the V4l2 capture driver will return 720x480 video size, but only odd lines have the video data, they are filled in line skip line mode.

2016-11-21 Update:

Added ISL79987 support, L3.10.53_ISL7998x_Surroundview_Patch_20161121.zip

New test capplication for de-interlance support: mxc_v4l2_tvin_isl7998x.tar.gz

Test commands (without de-interlace):

/mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 1 -x 0 -g2d &

/mxc_v4l2_tvin.out -ol 960 -ot 0 -ow 960 -oh 540 -d 1 -x 1 -g2d &

/mxc_v4l2_tvin.out -ol 0 -ot 540 -ow 960 -oh 540 -d 1 -x 2 -g2d &
/mxc_v4l2_tvin.out -ol 960 -ot 540 -ow 960 -oh 540 -d 1 -x 3 -g2d &

Test commands (with de-interlace, for ISL79987 only):

/mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 1 -x 0 -m 1 -g2d &
/mxc_v4l2_tvin.out -ol 960 -ot 0 -ow 960 -oh 540 -d 1 -x 1 -m 1 -g2d &
/mxc_v4l2_tvin.out -ol 0 -ot 540 -ow 960 -oh 540 -d 1 -x 2 -m 1 -g2d &
/mxc_v4l2_tvin.out -ol 960 -ot 540 -ow 960 -oh 540 -d 1 -x 3 -m 1 -g2d &

Now the same patch can support both ISL79985 and ISL79987, with NTSC CVBS camera, for ISL79985, it captures 60fps 720*240; for ISL79987, it captures 30fps 720*480.

2016-11-22 Update:

Added patch for L4.1.15 BSP, it supports both ISL79985 and ISL79987, L4.1.15_ISL7998x_Surroundview_Patch_20161122.zip

Test capplication mxc_v4l2_tvin_isl7998x.tar.gz is re-used.

[jsmith883]

Thanks Qiang_FSL‌, I had a couple different issues.  The main issue was I had my camera connected to the wrong pins on my board. *face palm*

The other issue I'm seeing is that the -g2d does not seem to be working.  In fact, after a couple of error messages, the system hangs and will not recover.  I then have to power cycle my board.  If I change it to -eglfs I do see the video on my LCD.  However, the video is kind of shaky - basically it looks like the video is moving left to right by about 5 pixels or so several times a second.  Here is the output:

# ./mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 1024 -oh 600 -d 1 -x 0 -g2d
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: mipi_lane_bps = 432 Mbps
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: value = 0xc.
g_in_width = 720, g_in_height = 240.
fb_fix.id = DISP3 FG.
fb: smem_start = 0x44700000, smem_len = 0x708000.
fb: frame buffer size = 0x258000 bytes.
fb: g_screen_info.xres = 1024, g_screen_info.yres = 600.
fb: g_display_left = 0.
fb: g_display_top = 0.
fb: g_display_width = 1024.
fb: g_display_height = 600.
start time = 1498795098 s, 51999 us
g2d_open: fail with status -7
Fail to open g2d device!
g2d_open: fail with status -7
Fail to open g2d device!
g2d_open: fail with status -7
Fail to open g2d device!
g2d_open: fail with status -7
Fail to open g2d device!
g2d_open: fail with status -7
Fail to open g2d device!
g2d_open: fail with status -7
Fail to open g2d device!
g2d_free: invalid g2d buf handle !
imx-ipuv3 2400000.ipu: IDMAC21's EBA0 is not 8-byte aligned
g2d_open: fail with status -7
Fail to open g2d device!
g2d_free: invalid g2d buf handle !
g2d_open: fail with status -7
Fail to open g2d device!
g2d_free: invalid g2d buf handle !
imx-ipuv3 2400000.ipu: IDMAC21's EBA0 is not 8-byte aligned

#### THE SYSTEM CRASHES AT THIS POINT AND WILL NOT RECOVER ####

#### IF I ADD -EGLFS, IT WORKS BUT VIDEO IS SHAKY ####

# ./mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 1024 -oh 600 -d 1 -x 0 -eglfs
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: mipi_lane_bps = 432 Mbps
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: value = 0xc.
g_in_width = 720, g_in_height = 240.
fb_fix.id = DISP3 FG.
fb: smem_start = 0x44700000, smem_len = 0x384000.
fb: frame buffer size = 0x12c000 bytes.
fb: g_screen_info.xres = 1024, g_screen_info.yres = 600.
fb: g_display_left = 0.
fb: g_display_top = 0.
fb: g_display_width = 1024.
fb: g_display_height = 600.
start time = 1498794277 s, 122146 us

Any thoughts on why -g2d does not work, or why the video would be shaky when I add -eglfs?

[joewu_msi]

Hi Qiang Li,

I can do the recording with the command "grecorder-1.0 --camera_id 0 --video_format 4 --width 720 --height 240 --fps 30".

However, I wonder how to perform recording in Qt5 application?

I try to invoke qt5 application "declarative-camera" located in /usr/share/qt5/examples/multimedia/declarative-camera, the precedure is as follows.

root@imx6qsabresd:~# cd /usr/share/qt5/examples/multimedia/declarative-camera

root@imx6qsabresd:~# export QT_QPA_PLATFORM=eglfs
root@imx6qsabresd:~# export QT_GSTREAMER_CAMERABIN_VIDEOSRC=imxv4l2src
root@imx6qsabresd:~# export QT_GSTREAMER_CAMERABIN_VIDEOSRC_DEVICE="/dev/video0"

root@imx6qsabresd:/usr/share/qt5/examples/multimedia/declarative-camera# ./declarative-camera

the qt5 program can be invoked, and I can see the preview video.

when pressing "Switch to Video" button, it occurs errors.

CameraBin error: "Internal data flow error."
CameraBin error: "Internal data flow error."

My hw board is "iMX6Q with ISL79985".

The BSP is Freescale Yocto "L4.1.15-2.0.0"

Could you give me some help/suggestion？

[timcui]

Hi Qiang 

   can this patch support iMX6S+isl79987? I don't find the ipu  module in the data sheet of imx6soloX Application Processor Reference Manual .

[qiang_li-mpu_se]

iMX6Solo has one IPU, so it can support up to two cameras with this patch.

iMX6SoloX has no IPU, so it can't use this patch.

[timcui]

Hi Qiang,

  Thanks  for your reply .We need four cameras input to ISL79987，So we should choose i.mx6D/Q that has two IPUs for this solution ?

[qiang_li-mpu_se]

Hi llinguo, this patch is for iMX6D/Q which has 2 IPU and can support up to 4 MIPI virtual channel.

In fact, ISL79987 can merge two cameras into one MIPI virtual channel, in this case, iMX6S/DL can also receive 4 cameras on two MIPI virtual channel, but we haven't implemented it, and no plan to support it.

[diguang]

Hi Qiang Li：

                    Thanks very much I We will use ISL79987  to 4 channel camera.   I think that  imx6Q/D  supports  max three channel camera before, I am not correct,. we use kernel 4.1.15 version also. So paches is very  important now,thanks.

[jiang_gs]

Hi qiang:
I use the demo which you provided, can display and switch between 0~3 camera.
But i hope to operate ISL79985 once, and display 4 camera to one screen.

Do ISL79985 has this function?
if any, Which register should I operate?

[qiang_li-mpu_se]

Hi Jiang xmen, you can create your own test application to get video from four cameras and render them to display once.

[timcui]

Hi Qaing

Our bsp version is 3.14.52,and the SOC is imx6DP,we apply the patch max9286 for 720P surround view .

Now we are using the main chip imx6solo  via bt656 interface for CVBS camera  surround view .The CVBS input chip is ISL79988.

So ,can  you give me the patch for bt656 input based on BSP version 3.14.52?

Or ,can you give me some suggestion to modify the kernel  for bt656 input?

Best Regards 

[rhqrhq]

HI ,Qiang Li.
我们的BSP版本是linux3.14.52,SOC为imx6dq,我们分别尝试了700 TV line、480 TV line、420 TV Line三种线数的PAL和NTSC制式的模拟摄像头， 发现除了480 tv line的摄像头是正常的之外，其余线数的摄像头出来的视频画面均有锯齿并且抖动的现象。不知是何原因,能不能给些建议？

非常感谢！！

[qiang_li-mpu_se]

700和420这2种都不是标准的CVBS摄像头，你可能要问一下Intersil那边，需要如何设置ISL78897，默认的代码。

Driver这边对于行数的修改，只要修改下面的参数就可以了：

unsigned int g_isl7998x_field_height = 240;
unsigned int g_isl7998x_frame_height = 480;

[nguyenmr]

Hi all,

I use ISL79985/87 and i have 4 inputs /dev/videox ( x = 0 - 3) and i tested camera streaming OK.

Now i want to porting them on Android M to use Android Camera API.

I tried to use Android Camera HAL in /hardware/imx/imx6/libcamera3 but not successful, when open Camera app, it shows can't init camera.

Does any one know how to do?

Thanks!

Nguyen

[nguyenmr]

I've fixed it. Now i can use camera by android camera app. However, the size is only 720x240, and the image is not stable, some time it's slicing from top to bottom. I don't know why! Maybe caused by noise on signal line or incorrect setting of registers of isl79987.

[jsmith883]

Hi Qiang Li - Mpu Se 

Are there any plans to migrate your patch to the 4.9 kernel?  Do you already have a solution for this?

[jiang_gs]

Hi Qiang Li, 
I use imx6Q+isl79987 kernel 3.14.52 , and i capture in 720*480,
I use L3.10.53_ISL7998x_Surroundview_Patch_20161121 patch to edit kernel, and test ok!

I found a problem， when I test video0,video1,video2,video, open is ok!

but video output image is always scrolling.
can you tell me what can I do?

thank you !

[qiang_li-mpu_se]

Maybe you are using PAL cameras, this patch is set to NTSC camera in default. You can check it.

[jiang_gs]

Hi Qiang:

I have test PAL and NTSC fmt, but it not work ok.

and then I debug in kernel, I found there is Tow IPU in IMX6Q, 

and  video0, video1  used ipu0,     video2, video3  used ipu1, 

video0 , video2 : CSI  IC  MEM;  (ipu_prp_enc.c)

video1 , video3 : CSI   MEM;   (ipu_csi_enc.c)

The result is video0, video2 is work OK.

BUT video1 , video3 is always scrolling.

It's very strange.  

Can you tell me what can I do ?

Thank You Very Match!!!

the picture is video0, video1,  Left is video0, it work ok,  Right is video1, it scrolling.

[qiang_li-mpu_se]

Hi Jiang xmen, I think you should reference to the mxc_v4l2_tvin test application in this document, for this use case, all cameras should use CSI->MEM path to capture.

[mike_sims]

Hi,

I duplicated the initial steps of this article using NXP's 4.1.15 kernel on an i.MX6D integrating the above patches, but we have our own unique IMX6D Carrier board.   This board only has video capture of the ISL79987 connected to the MIPI port using 2 data lanes (exactly like described above), so I feel we should be clean on our implementation  (very similar to SabreSD), but when I power it up I see this: 

As you can see, it is an image replicated 8 times on the screen with no / bad color...  

Each of the smaller images looks to be correct.   The camera I am using has IR Lighting which is why there is a bright image in the middle, but other than this, the Black & White component of the video look OK.

I have to admit I am completely new to MIPI, so at this point, I am not sure where to start my search...   I have used the ISL TW Terminal to ensure the input decoder is working correctly, and it is.   I have enabled the On Chip Pattern Generator, but it too is displayed as 8 smaller images.  I have reviewed my configuration against another post on this site with regard to verifying MIPI configurations, and it seems I have things correct, but still I have this issue.

Does someone recognize the above image and a possible solution?   Of if not, how about details of setup & diagnostics of a multi camera MIPI configuration like the ISL79987?

Many thanks in advance...

Best Regards,
Mike Sims

[nguyenmr]

Hi Jiang,

did you fix this problem? i faced the same problem: video's crolling on both video0/video1. i'm using isl79987 + kernel 4.1.15, CSI->MEM path. I tried to use CSI->IC->MEM but it cannot stream.

[jiang_gs]

I  use mxc_v4l2_tvin demo, test ok.

[davood]

Is there a design file for any iMx6/8 using ISL79987 ?

[zbunix]

HI ,Qiang Li：
     (1) 同时使用4路TVIN(2xCSI, 2xMIPI)，用了3颗AD芯片连接到imx6d三个camera接口上,

     

     (2) IMX6D+L3.10.53+L3.10.53_ISL7998x_Surroundview_Patch_20161121 patch.

     (3) GPR1配置： bit19 is 1, bit20 is 1

     发现问题parallel0和parallel1正常，CVBS_PAL0和CVBS_PAL1不正常，不知是何原因,能不能给些建议？

非常感谢！！

[qiang_li-mpu_se]

CVBS_PAL0需要从MIPI的Virtual channel 1去抓取，CVBS_PAL1需要从MIPI的Virtual channel 2去抓取，MIPI camera驱动需要实现好。

[dioneva_k]

Hi, I am using ISL79987 and need to run as MIPI CSI2. Look at the kernel startup:

[    0.250848] mxc_mipi_csi2 21dc000.mipi_csi: i.MX MIPI CSI2 driver probed ipu0 csi
[    0.250861] mxc_mipi_csi2 21dc000.mipi_csi: i.MX MIPI CSI2 dphy version is 0x3130302a
[    1.795855] mxc_v4l2_master_attach: ipu(0:0)/csi(0:1)/mipi(0:0) doesn't match
[    1.795873] mxc_v4l2_master_attach: ipu0:/csi0 parallel attached isl7998x:mxc_v4l2_cap0
[    1.795891] mxc_v4l2_master_attach: ipu0:/csi1 parallel attached isl7998x:mxc_v4l2_cap1
[    1.795902] mxc_v4l2_master_attach: ipu(1:0)/csi(0:1)/mipi(0:0) doesn't match
[    1.795908] mxc_v4l2_master_attach: ipu(1:0)/csi(0:0)/mipi(0:0) doesn't match
[    1.795918] mxc_v4l2_master_attach: ipu(1:0)/csi(1:1)/mipi(0:0) doesn't match
[    1.795924] mxc_v4l2_master_attach: ipu(1:0)/csi(0:1)/mipi(0:0) doesn't match
[    1.795928] mxc_v4l2_master_attach: ipu(1:0)/csi(1:0)/mipi(0:0) doesn't match
[    1.795934] mxc_v4l2_master_attach: ipu(1:0)/csi(0:0)/mipi(0:0) doesn't match
[    1.795939] isl7998x_mipi is found

Running these commands:

# gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! autovideosink
# gst-launch-1.0 v4l2src device=/dev/video1 ! videoconvert ! autovideosink

I have this result:

[  506.984147] mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: mipi_lane_bps = 432 Mbps
[  506.991994] mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: value = 0xc.
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[  507.664909] UYVY
[  518.233107] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate a buffer
Additional debug info:
../../../git/sys/v4l2/gstv4l2src.c(866): gst_v4l2src_create (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0
Execution ended after 0:00:10.610824001
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Can anybody help me?

[dioneva_k]

Hi, I am using ISL79987 and need to run as MIPI CSI2.

The virtual channels 1 and 2 are working, but the virtual channels 0 and 3 aren't working.

Channels 0 and 3 return this error:

# ./mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 1280 -oh 720 -d 1 -x 0 -g2d

g_in_width = 720, g_in_height = 480.
fb_fix.id = DISP3 FG.
fb: smem_start = 0x85000000, smem_len = 0x546000.
fb: frame buffer size = 0x1c2000 bytes.
fb: g_screen_info.xres = 1280, g_screen_info.yres = 720.
fb: g_display_left = 0.
fb: g_display_top = 0.
fb: g_display_width = 1280.
fb: g_display_height = 720.
start time = 1545671531 s, 683461 us
[ 379.992842] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0
VIDIOC_DQBUF failed.

Can anybody help me?

[qiang_li-mpu_se]

Hi Dionevã Krolow, please make sure you had disabled other camera drivers (such as ov5640) from device tree and kernel config files. Another thing can be checked is the IOMUX_GPR1 register setting, BIT 20 and BIT 19 should be 0 to map MIPI VC 0 and 3 to IPU CSI port.

The code is in arch\arm\mach-imx\mach-imx6q.c, function imx6q_csi_mux_init().

[dioneva_k]

Hi Qiang Li - Mpu Se, it is working thank you very much!!!!! :smileygrin: :smileygrin: :smileygrin:

I would like to know how to remove the borders to put the video in full screen.
 In 720x480 resolution appear small black edges in Qt5 QML Camera.
 Is it possible to remove them?

[qiang_li-mpu_se]

Hi Dionevã Krolow ,

The IPU CSI module can do crop as you wanted. For example, the original setting to capture 720*480, in register IPUx_CSIx_ACT_FRM_SIZE, CSIx_ACT_FRM_WIDTH = (720-1) and CSIx_ACT_FRM_HEIGHT = (480-1). If we set CSIx_ACT_FRM_WIDTH to (712-1), then the last 8 pixel of each line dropped, you can capture 712*480 pictures.

And if you set  CSIx_ACT_FRM_WIDTH to (712-1), and in register IPUx_CSIx_OUT_FRM_CTRL, set CSIx_HSC to 8, then the first 8 pixel of each line dropped, you can still capture 712*480 pictures.

Note, in all above setting, the IPUx_CSIx_SENS_FRM_SIZE register setting shoule not be changed, it needs keep the value as capture 720*480 pictures, but the IDMAC setting should also be changed to 712*480 resolution.

[dioneva_k]

Hi Qiang Li, thank you very much!!! :smileygrin:

Now I have the following problem:
imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00000001
But I do not have this problem always, it's random, before and after this modification

Is it possible to remove this error?

[davood]

Should there be a driver for the iMX8M8 as well?

[dioneva_k]

Does this driver support cameras with a resolution of 1280x960?

Cameras with a resolution of 960x540 are working, but another camera with a resolution of 1280x960 does not work. Screen is blue permanently.

Can anybody help me?

[tenrisabo]

Low power, high performance. Simplify development with IP reference designs. Comprehensive Library. Integrated Flash Memory. LiteBlue