iMX6DQ MAX9286 MIPI CSI2 720P camera surround view solution for Linux BSP

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

iMX6DQ MAX9286 MIPI CSI2 720P camera surround view solution for Linux BSP

No ratings

iMX6DQ MAX9286 MIPI CSI2 720P camera surround view solution for Linux BSP

For iMX6DQ, there are two IPUs, so they can support up to 4 cameras at the same time. But the default BSP can only support up to two cameras at the same time.

87418_87418.pnguntitled.png

The attached patch can make the BSP support up to 4 cameras based on 3.10.53 GA 1.1.0 BSP.

 

The 4 cameras can be:

- 1xCSI, 3xMIPI

- 2xCSI, 2xMIPI

- 4xMIPI

 

For 4xMIPI case, the four cameras should be combined on the single MIPI CSI2 interface, and each camera data should be transfered on a mipi virtual channel.

 

In this patch, we given the example driver for Maxim MAX9286, it was verified working on iMX6DQ SabreAuto board. The input to MAX9286 is four 720P30 cameras.

The verified camera boards:

    (1) Onsemi AR0140+AP0101+MAX9271 boards.

    (2) OmniVision OV10635+MAX9271 boards.

 

The MIPI CSI2 CVBS camera surround view solution can be found at: iMX6DQ ISL79985/79987 MIPI CSI2 CVBS camera surround view solution for Linux BSP

The MIPI CSI2 CVBS HD camera surround view solution can be found at: iMX6DQ TP2854 MIPI CSI2 720P CVBS camera surround view solution for Linux BSP

 

The kernel patches:
0001-IPU-update-IPU-capture-driver-to-support-up-to-four-.patch
     Updated IPU common code to support up to four cameras.

 

0002-Add-Max9286-support-on-SabreAuto-board-which-can-sup.patch
     MAX9286 driver, it includes MAX9271, AP0101 and AR0140 drivers.

 

0003-Remove-the-page-size-align-requirement-for-v4l2-capt.patch
     With this patch, the mxc_v4l2_tvin test application can use overlay framebuffer as V4l2 capture buffer directly.

 

0004-Max9286-skip-AP0101-camera-re-initialization.patch
     If the camera board's power had been kept after initialized, this patch will bypass the re-initialization to reduce the start up time.

 

0005-Max9286-set-I2C-speed-to-400Kbps.patch
    Set I2C to 400Kbps to reduce the AP0101+AR0140 initialization time.

 

0006-Max9286-add-retry-for-MAX9271-I2C-access.patch
    Added retry for MAX9271 I2C access.

 

0007-Max9286-Add-support-for-OV10635-camera.patch
    Updated code for OV10635 camera.

 

0008-Max9286-support-auto-detect-camera-number.patch
    Make the Max9286 driver can detect the camera number automatically.

 

 

How to builld the kernel with MAX9286 support:

      make imx_v7_defconfig

      make menuconfig (In this command, you should select the MAX9286 driver:

            Device Drivers  --->

                  <*> Multimedia support  --->

                        [*]   V4L platform devices  --->

                              <*>   MXC Video For Linux Video Capture

                                      MXC Camera/V4L2 PRP Features support  --->

                                          <*>Maxim max9286 GMSL Deserializer Input support

                                              Select Camera Sensor (OmniVision OV10635 camera sensor)  // Or (Onsemi AP0101 and AR0140 camera sensor)

                                          <*>mxc VADC support

                                          <*>Select Overlay Rounting (Queue ipu device for overlay library)

                                          <*>Pre-processor Encoder library

                                          <*>IPU CSI Encoder library)

      make zImage

      make dtbs

 

The built out image file:

      arch/arm/boot/dts/imx6q-sabreauto.dtb

      arch/arm/boot/zImage

 

"mxc_v4l2_tvin_max9286.tar.gz" is the test application, test command to capture the four cameras and render on 1080P HDMI display:

/mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 1 -x 0 -g2d &

/mxc_v4l2_tvin.out -ol 960 -ot 0 -ow 960 -oh 540 -d 1 -x 1 -g2d &

/mxc_v4l2_tvin.out -ol 0 -ot 540 -ow 960 -oh 540 -d 1 -x 2 -g2d &

/mxc_v4l2_tvin.out -ol 960 -ot 540 -ow 960 -oh 540 -d 1 -x 3 -g2d &

 

Some hardware check point on AR0140+AP0101+MAX9271 camera board (Please get MAX9286 and OV10635 schematics from Maxim):

1. In this patch, MAX9286's I2C address is 0x4D, so ADD0 and ADD1 should be connected to high.

AP0101's I2C address is 0xBA, so SADDR should be connected to high.

 

2. AP0101's DOUT0~DOUT7 should be connected to MAX9271's DIN7~DIN0, the order should be switched, MSB connected to LSB.

 

3. MAX9271's GPO pin should be connected to AP0101's FRAME_SYNC pin. The pull down resistance on FRAME_SYNC pin should not be 0 ohm.

 

Some known limitation:

1. AP0101's VSYNC invalid time, last video line's HSYNC to VSYNC porch's max value is 255 pixel clocks, it is not enough for MAX9286 to generate the Frame End MIPI packets for each camera. So in order to let iMX6DQ to capture 1280x720 video for each camera, we had let AP0101 output 1280*724 frame size, and iMX6 will only capture 720 lines, the remained video data and Frame End will be ignored.

This solution will not impact the function, but there will be "Error matching Frame Start with Frame End for Virtual Channel x" error reported from iMX6 MIPI_CSI_ERR1 register.

Maxim suggested to use MAX96705 to relace the MAX9271, it can delay the VSYNC invalid time, then the MIPI error will be fixed.

 

 

2015-11-17 update: Updated for OV10635 camera support.

File: L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2015-11-17.zip

 

2015-12-04 update:

File: L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2015-12-04.zip

Added patch 0009-Max9286-updated-PCLK-edge-setting-for-OV10635.patch to correct the OV10635 PCLK edge setting

 

 

2016-03-07 update:

File L3.14.38_GA_MAX9286_Surroundview_Patch_2016-03-07.zip

Added kernel patch for L3.14.38 GA 1.1.0 BSP.

 

2016-07-26 update:

Files: L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip; L3.14.38_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip; L3.14.52_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip.

Added gstreamer support.

Added MAX96705 support.

Added patch for L3.14.52_GA1.1.0.

 

2017-12-11 update:

Added CVBS surround view link: iMX6DQ TP2854 MIPI CSI2 720P CVBS camera surround view solution for Linux BSP

 

 

2021-04-26 update:

Some customer reported, when system loading is heavy, sometimes, some camera will flicker left and right. It is caused by SFMC FIFO data lost. The original patch used IDMAC 0 and IDMAC 1 for two cameras on one IPU, this is not the best setting. 

IDMAC 1 is fixed to use 1/4 SMFC FIFO and it will cause IDMAC 0 to use 1/4 SMFC FIFO too. And another 1/2 of SMFC FIFO can't be used in this case.

qiang_li-mpu_se_0-1619426236653.png

Some code update to improve it:

For each IPU, please use IDMAC 0 and IDMAC 2 to capture the two cameras.
This needs change the hard coding in "drivers\media\platform\mxc\capture\ipu_csi_enc.c",
"CSI_MEM1" and "IPU_IRQ_CSI1_OUT_EOF" should be changed to "CSI_MEM2" and "IPU_IRQ_CSI2_OUT_EOF". In this case, all SMFC FIFO can be used.

And in "ipu_common.c", function ipu_probe(), the followed code should be changed to make IDMAC2 use high priority too.
/* Set sync refresh channels and CSI->mem channel as high priority */
- ipu_idmac_write(ipu, 0x18800003L, IDMAC_CHA_PRI(0));
+ ipu_idmac_write(ipu, 0x1880000FL, IDMAC_CHA_PRI(0));

Attachments
Comments

Hi Qiang Li:

I use the IMX6Q core board. Now when using both GC2000 and GC320, there are occasional image tears on the screen display.
The specific implementation is like this:
   Get the real-time video data of the 4-way camera, display it on the GPU and display it in real time on the lvds screen. At the same time, the 4 channels of video are called by g2d to call the g2g_blit function for resize. The image of each frame is 720P resize to 640*480, and then synthesized into one frame. Image so that 4 channels of video can be displayed in one frame.the spliced video is not display on the screen and saved directly into the video,but now it is found that the video displayed by the GPU rendering in real time will be torn. Playback of the saved video also found that the image has a tearing problem, affecting the viewing effect.
   If you test the GPU rendering display separately, or the video of 4 channels of video is saved by G2D synthesis, there will be no image tearing problem.
But as long as these two functions are combined into one project, the problem of image tearing will occur when displaying and the saving  video.
1. Can I use the GC2000 of IMX6 and GC320 at the same time? Is there a problem with GPU resource preemption?
2. Is there a problem with data conflicts in the GPU? I have already made the GC2000 and GC320 get data that cannot be bufed, but after the two functions are integrated, there will still be a problem of video tearing.
I have been plagued by this problem for several weeks, and there has been no good solution. I am very anxious.

Hi Qiang Li:

I use the IMX6Q core board. Now when using both GC2000 and GC320, there are occasional image tears on the screen display.
The specific implementation is like this:
   Get the real-time video data of the 4-way camera, display it on the GPU and display it in real time on the lvds screen. At the same time, the 4 channels of video are called by g2d to call the g2g_blit function for resize. The image of each frame is 720P resize to 640*480, and then synthesized into one frame. Image so that 4 channels of video can be displayed in one frame.the spliced video is not display on the screen and saved directly into the video,but now it is found that the video displayed by the GPU rendering in real time will be torn. Playback of the saved video also found that the image has a tearing problem, affecting the viewing effect.
   If you test the GPU rendering display separately, or the video of 4 channels of video is saved by G2D synthesis, there will be no image tearing problem.
But as long as these two functions are combined into one project, the problem of image tearing will occur when displaying and the saving  video.
1. Can I use the GC2000 of IMX6 and GC320 at the same time? Is there a problem with GPU resource preemption?
2. Is there a problem with data conflicts in the GPU? I have already made the GC2000 and GC320 get data that cannot be bufed, but after the two functions are integrated, there will still be a problem of video tearing.
I have been plagued by this problem for several weeks, and there has been no good solution. I am very anxious.

Hi, liqiang

In my local environment which using immunity mode, and do data line swap according to your suggestion,

which causes image color change but problem NOT solved.

(帖子里好像帖不了附件,单独像您提了一个问题。)

By the way, I am using leopardimaging LI-USB30-OV10635-SER camera as following:

https://www.leopardimaging.com/uploads/LI-OV10635-SER_datasheet.pdf

It works well when connecting to PC by USB 3.0.

please help check, thanks!

Hi,liqiang

I have solve the color fault problem, by swap data line using ov10635 register 0x4709, so thank you very much!!

This had been explained, g2d_blit target buffer should be the current displayed framebuffer:

"

The current mxc_v4l2_tvin demo application will render camera video to current frame buffer directly, so there will be tearing issue.

For real use case, you need use multi-buffers, for example, the virtual framebuffer size is 3 times of real display buffer, and each time only one buffer is showing on display, so when buffer 1 is showing on display, you can draw combined camera videos into buffer 2, after finished drawing, call ioctl FBIOPAN_DISPLAY to show buffer 2 on screen.

"

The color issue happens between MAX96705 and MAX9286 mapping, so PC USB has no use for reference.

There are several kind of mapping order on data line, 8 bits/10bits, MSB/LSB. You need fine tuning it based on the OV10635 and MAX96705 connecting.

Qiang_FSL

Hi, Liqiang

Could you please tell me what's the fps value

if we run the surroundview(4 camera 1280x720) application base your patch and  Imx6d processor?

whether he DDR bandwidth is enough? thank you!

以下中文:

-----------------

如果使用这个patch并跑在imx6d的板子上,目前surroundview(基于4个720p的摄像头)能达到多少个fps,

内存带宽够用吗?谢谢!

---------------

Hi z dq,

Did you solve the instability problem? I got sensor clk error when connecting more than 2 cameras although MAX96705's GPO pin are connected to AP0101's FRAME_SYNC pin. 

I rarely got the error with 2 cameras .

28.463487] max9286_mipi 1-006a: ap0101_initialize: AP0101 hasn't been found, reg = 0x0, index = 4.
[ 28.472625] max9286_mipi 1-006a: max9286_write_reg:write reg error:reg=34,val=b6
[ 28.673495] max9286_mipi 1-006a: max9271_write_reg:write reg error:reg= 4,val=83
[ 28.783553] max9286_mipi 1-006a: max9286_write_reg:write reg error:reg=15,val=9b
[ 29.403469] mipi csi2 can not receive sensor clk! MIPI_CSI_PHY_STATE = 0x6f0.

Regards

您好,能否请教个问题:1.我看FH8536好像只有模拟输出,您是怎么跟96705连接的;2.用IMX225+FH8536的话,四个镜头模组的图像如何同步?盼复,感谢!

96705有个GPO管脚,可以输出一个同步30HZ triger信号,可以直接接sensor的一个triger in 一起来同步。同时sensor 工作在slave模式下。

dezhijiang@126.com

谢谢回复,我们现在用0143+0101+96705已经正常跑起来,0143的同步确实是sensor的trigger in来控制,但是IMX225貌似没看到有trigger这个pin,datasheet里面描述slave mode是需要提供VS和HS;另外,我看FH8536这个ISP只有模拟输出,96705好像也没法接模拟输入,不清楚IMX225+FH8536+MAX96705这个方案是怎么解决这两个问题的

Dear Qiang Li,

        Sorry, I am some confused, as you said, Imx6Q can support up to 4 cameras, 

The 4 cameras can be:

- 1xCSI, 3xMIPI

- 2xCSI, 2xMIPI

- 4xMIPI

what is the meaning of  1x CSI and 3x MIPI, etc? usually we say MIPI CSI-2 together,  CSI is the same thing as MIPI CSI-2?

BTW, how many BT656 interfaces can be directly supported by Imx6Q simultaneously?  

  

Many thanks~~

Each IPU has two CSI modules, their input can be parallel CSI interface or MIPI CSI2 virtual channel. And for iMX6DQ, it can support up to two parallel CSI interfaces (BT656 interface). Both of them can work at the same time.

Thanks Qiang!

So can we have 2 x BT656 and 2 x MIPI CSI2 work together?  if yes , how many virtual channels(1~4) can each of the 2x MIPI CSI2 have? 

2 x MIPI CSI2 means two MIPI virtual channels(VC1 and VC2) on one physical MIPI CSI2 interface.

Thanks a lot! Qiang Li. 

Understand now.

你好,Qiang_FSL 

目前我使用了adv7181d (格式:BT1120)和nextchip(N4)这两个IC

adv7180配置如下:

v4l2_cap_0 {
compatible = "fsl,imx6q-v4l2-capture";
      ipu_id = <0>;
      csi_id = <0>;
      mclk_source = <0>;
      status = "okay";
};

adv7180: adv7180@21 {
      compatible = "adv,adv7180";
      reg = <0x21>;
      pinctrl-names = "default";
      pinctrl-0 = <&pinctrl_ipu1_2>;
      clocks = <&clks IMX6QDL_CLK_CKO>;
      clock-names = "csi_mclk";
      pwn-gpios = <&gpio1 4 0>;
      reset-gpios = <&gpio3 16 GPIO_ACTIVE_HIGH>;
      ipu_id = <0>;
      csi_id = <0>;
      mclk = <24000000>;
      mclk_source = <0>;
      cvbs = <1>;
};

nextchip(N4)本身可以接四个摄像头,但是我仅使用了 的的其中两路,分辨率是720p,然后配置如下:

mipi-csi2-channel@2 {
      ipu_id = <1>;
      csi_id = <0>;
      v_channel = <2>;
      status = "okay";
};

mipi-csi2-channel@3 {
      ipu_id = <1>;
      csi_id = <1>;
      v_channel = <3>;
      status = "okay";
};

----------------------------------------------------------------------

v4l2_cap_1 {
      compatible = "fsl,imx6q-v4l2-capture";
      ipu_id = <1>;
      csi_id = <0>;
      mclk_source = <0>;
      status = "okay";
};

v4l2_cap_2 {
compatible = "fsl,imx6q-v4l2-capture";
      ipu_id = <1>;
      csi_id = <1>;
      mclk_source = <0>;
      status = "okay";
};

目前三个摄像头都可以采到图像,但是 ipu_id = <1>;   csi_id = <1>;这一路的摄像头采集到的图像会闪动,看起来就像是图像断层,我用mxc-v4l2-tvin去打开不会闪

mxc-v4l2-tvin  -ol 0 -ot 0 -ow 1280 -oh 800 -d 1 -x 2 

我调通了HAL层,我们应用给我写了一个demo去分别打开每个video,当打开video2( ipu_id = <1>;   csi_id = <1>;)的时候就出现闪动的情况,目前不知道从哪里分析,可否提供一点思路,谢谢!

我拍照的时候拍不到闪动断层的现象,我录制了一个MP4格式视频,但是在上传过程中提示格式不支持,请问我该上传什么格式的视频?

Excuse me, can I have another patch of Linux 3.14.38? That link failed

HI Liqiang:

您好!

  我这边使用imx6d + max9286 + max96705的方式进行三路camera同时预览。当接入2个或者三个camera的时候,预览画面就会有抖动,滚动现象出现;

我只接一个摄像头是正常的。

跟美信fae这边确认camera都是frame sync同步模式。

这个问题查了好久,麻烦帮忙看下

Hi 佳杰 吴 

要看你用的什么camera sensor,如果买的现成的模组,一般都是带flash的,那么它的工作模式就是在flash ROM里面设置的,MAX9286这边即使设置了Frame Sync模式,也是没法改变camera的模式的。

另外有些camera sensor(比如AP0101+AR0130),最后一行的HSYNC to VSYNC 的时间少于255个pixel clock,也是会有问题的,这个时候需要让IPU CSI那边少抓几行数据,比如720行的输入,IPU CSI只抓712行。

我这边使用的感光sensors是AP0143;sensor这边确定已经做frame sync,因为之前的模式,多路sensor不能出图;

现在我抓320行,接三个sensor的时候,不会出现滚动,但是会出现横条,抖动现象。

你可以测量一下MAX9286的lock pin,它应该要一直是high,才能保证MIPI的数据是稳定输出的,如果camera和MAX9286没有设置好,就不能保持一直高电平,这样数据就不会完整。

从之前AP0101的测试结果看,MAX9286要设成manual模式,然后0x08那个寄存器,是要根据camera的设置帧率,做调整的:

#ifdef SENSOR_AP0101
 //Manual Mode
 max9286_write_reg(0x01, 0x00);

 max9286_write_reg(0x63, 0x00);
 max9286_write_reg(0x64, 0x00);

 max9286_write_reg(0x06, 0x00);
 max9286_write_reg(0x07, 0x00);
 max9286_write_reg(0x08, 0x26);
#endif

现在量了这个lock引脚一直都是低的,接一个sensor的时候也是低的;

max9286的模式已经改过,那几个参数也都调整过,但是没有效果;

现在想排除下是不是imx6 这边出了问题,该如何排查?

MAX9286 的lock pin不能保持高电平,表示它不能完全锁住camera信号,那么MIPI CSI2上的数据就一定是有问题的,这个和iMX6这边无关。

这个脚电平是低,但是读了下0x27寄存器的bit7却是1 :

pastedImage_1.png

这个是不是也不正常。

Hi Qiang Li,

I am using Imx6qp. With following commands, the system is working properly.

WIDTH=640
HEIGHT=240
./mxc_v4l2_tvin.out -ol 0 -ot 0 -ow ${WIDTH} -oh ${HEIGHT} -d 0 -x 0 -g2d &
./mxc_v4l2_tvin.out -ol ${WIDTH} -ot 0 -ow ${WIDTH} -oh ${HEIGHT} -d 0 -x 1 -g2d &
./mxc_v4l2_tvin.out -ol 0 -ot ${HEIGHT} -ow ${WIDTH} -oh ${HEIGHT} -d 0 -x 2 -g2d &
./mxc_v4l2_tvin.out -ol ${WIDTH} -ot ${HEIGHT} -ow ${WIDTH} -oh ${HEIGHT} -d 0 -x 3 -g2d &

pastedImage_1.jpg

When I try to use gstreamer with the following pipelines, third and forth cameras are jittering and scrolling.

WINDOW1="window-x-coord=0 window-y-coord=0 window-width=${WIDTH} window-height=${HEIGHT}"
WINDOW2="window-x-coord=${WIDTH} window-y-coord=0 window-width=${WIDTH} window-height=${HEIGHT}"
WINDOW3="window-x-coord=0 window-y-coord=${HEIGHT} window-width=${WIDTH} window-height=${HEIGHT}"
WINDOW4="window-x-coord=${WIDTH} window-y-coord=${HEIGHT} window-width=${WIDTH} window-height=${HEIGHT}"

gst-launch-1.0 -v -e imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 force-aspect-ratio=false $WINDOW1 &
gst-launch-1.0 -v -e imxv4l2videosrc device=/dev/video1 ! imxg2dvideosink framebuffer=/dev/fb0 force-aspect-ratio=false $WINDOW2 &
gst-launch-1.0 -v -e imxv4l2videosrc device=/dev/video2 ! imxg2dvideosink framebuffer=/dev/fb0 force-aspect-ratio=false $WINDOW3 &
gst-launch-1.0 -v -e imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink framebuffer=/dev/fb0 force-aspect-ratio=false $WINDOW4 &

pastedImage_1.gif

v4l applications are working properly but gstreamer causes the problem seen above. Do you have any idea about this problem?

For gstreamer, when you render 4 cameras together, it needs combine the buffer 4 times and swap the buffer 4 times, this will impact the performance. For tvin test application, it just renders the cameras to target framebuffer directly, no buffer swap, this can cause sawtooth issue.

The best way for rendering: draw the 4 cameras to one target buffer, then swap it to frambuffer in one time.

hi liqiang:

   我现在在android6系统上面进行多路camera同时录像,用mMediaRecorder进行录制;

   录制一路的时候没有问题,但是录制两路的时候gpu报错:

pastedImage_1.png

pastedImage_2.png

   这个是什么原因?

Hi Qiang Li,

The following two pipelines work normally when run seperately.

  • gst-launch-1.0 -v -e imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 force-aspect-ratio=false 
  • gst-launch-1.0 -v -e imxv4l2videosrc device=/dev/video1 ! imxg2dvideosink framebuffer=/dev/fb0 force-aspect-ratio=false 

However, each pipelines given below (especially with video3) cause sawtooth issue given as attached gif, even if they run seperately.

  • gst-launch-1.0 -v -e imxv4l2videosrc device=/dev/video2 ! imxg2dvideosink framebuffer=/dev/fb0 force-aspect-ratio=false 
  • gst-launch-1.0 -v -e imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink framebuffer=/dev/fb0 force-aspect-ratio=false 

pastedImage_2.gif

While twin app does not create any problem for each camera.

While using gstreamer, what can cause this situation? Is it about gstreamer patch or IPU2?

The log shows there is no enough memory, I think you need reserve more memory for GPU.

As I explained, this is the render issue, for 60fps display, the PAN display can only be up to 60fps, but for 4x720P30 camera, if you render them alone, there are total 120fps, so it can't work correctly. You need combine the 4 cameras together then render them once.

Hi Qiang Li,
 
Sorry for the confusion. I assumed that below single pipe which causes saw-tooth issue does not render all cameras.
 
#gst-launch-1.0 -v -e imxv4l2videosrc device=/dev/video2 ! imxg2dvideosink framebuffer=/dev/fb0 force-aspect-ratio=false
 
I am studying about gstreamer to draw 4 cameras to one target buffer.
 
Thanks and best regards.
Version history
Last update:
‎04-26-2021 01:39 AM
Updated by: