iMX6DQ MAX9286 MIPI CSI2 720P camera surround view solution for Linux BSP

[#1]

For iMX6DQ, there are two IPUs, so they can support up to 4 cameras at the same time. But the default BSP can only support up to two cameras at the same time.

The attached patch can make the BSP support up to 4 cameras based on 3.10.53 GA 1.1.0 BSP.

The 4 cameras can be:

- 1xCSI, 3xMIPI

- 2xCSI, 2xMIPI

- 4xMIPI

For 4xMIPI case, the four cameras should be combined on the single MIPI CSI2 interface, and each camera data should be transfered on a mipi virtual channel.

In this patch, we given the example driver for Maxim MAX9286, it was verified working on iMX6DQ SabreAuto board. The input to MAX9286 is four 720P30 cameras.

The verified camera boards:

    (1) Onsemi AR0140+AP0101+MAX9271 boards.

    (2) OmniVision OV10635+MAX9271 boards.

The MIPI CSI2 CVBS camera surround view solution can be found at: iMX6DQ ISL79985/79987 MIPI CSI2 CVBS camera surround view solution for Linux BSP

The MIPI CSI2 CVBS HD camera surround view solution can be found at: iMX6DQ TP2854 MIPI CSI2 720P CVBS camera surround view solution for Linux BSP

The kernel patches:
0001-IPU-update-IPU-capture-driver-to-support-up-to-four-.patch
     Updated IPU common code to support up to four cameras.

0002-Add-Max9286-support-on-SabreAuto-board-which-can-sup.patch
     MAX9286 driver, it includes MAX9271, AP0101 and AR0140 drivers.

0003-Remove-the-page-size-align-requirement-for-v4l2-capt.patch
     With this patch, the mxc_v4l2_tvin test application can use overlay framebuffer as V4l2 capture buffer directly.

0004-Max9286-skip-AP0101-camera-re-initialization.patch
     If the camera board's power had been kept after initialized, this patch will bypass the re-initialization to reduce the start up time.

0005-Max9286-set-I2C-speed-to-400Kbps.patch
    Set I2C to 400Kbps to reduce the AP0101+AR0140 initialization time.

0006-Max9286-add-retry-for-MAX9271-I2C-access.patch
    Added retry for MAX9271 I2C access.

0007-Max9286-Add-support-for-OV10635-camera.patch
    Updated code for OV10635 camera.

0008-Max9286-support-auto-detect-camera-number.patch
    Make the Max9286 driver can detect the camera number automatically.

How to builld the kernel with MAX9286 support:

      make imx_v7_defconfig

      make menuconfig (In this command, you should select the MAX9286 driver:

            Device Drivers  --->

                  <*> Multimedia support  --->

                        [*]   V4L platform devices  --->

                              <*>   MXC Video For Linux Video Capture

                                      MXC Camera/V4L2 PRP Features support  --->

                                          <*>Maxim max9286 GMSL Deserializer Input support

                                              Select Camera Sensor (OmniVision OV10635 camera sensor)  // Or (Onsemi AP0101 and AR0140 camera sensor)

                                          <*>mxc VADC support

                                          <*>Select Overlay Rounting (Queue ipu device for overlay library)

                                          <*>Pre-processor Encoder library

                                          <*>IPU CSI Encoder library)

      make zImage

      make dtbs

The built out image file:

      arch/arm/boot/dts/imx6q-sabreauto.dtb

      arch/arm/boot/zImage

"mxc_v4l2_tvin_max9286.tar.gz" is the test application, test command to capture the four cameras and render on 1080P HDMI display:

/mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 1 -x 0 -g2d &

/mxc_v4l2_tvin.out -ol 960 -ot 0 -ow 960 -oh 540 -d 1 -x 1 -g2d &

/mxc_v4l2_tvin.out -ol 0 -ot 540 -ow 960 -oh 540 -d 1 -x 2 -g2d &

/mxc_v4l2_tvin.out -ol 960 -ot 540 -ow 960 -oh 540 -d 1 -x 3 -g2d &

Some hardware check point on AR0140+AP0101+MAX9271 camera board (Please get MAX9286 and OV10635 schematics from Maxim):

1. In this patch, MAX9286's I2C address is 0x4D, so ADD0 and ADD1 should be connected to high.

AP0101's I2C address is 0xBA, so SADDR should be connected to high.

2. AP0101's DOUT0~DOUT7 should be connected to MAX9271's DIN7~DIN0, the order should be switched, MSB connected to LSB.

3. MAX9271's GPO pin should be connected to AP0101's FRAME_SYNC pin. The pull down resistance on FRAME_SYNC pin should not be 0 ohm.

Some known limitation:

1. AP0101's VSYNC invalid time, last video line's HSYNC to VSYNC porch's max value is 255 pixel clocks, it is not enough for MAX9286 to generate the Frame End MIPI packets for each camera. So in order to let iMX6DQ to capture 1280x720 video for each camera, we had let AP0101 output 1280*724 frame size, and iMX6 will only capture 720 lines, the remained video data and Frame End will be ignored.

This solution will not impact the function, but there will be "Error matching Frame Start with Frame End for Virtual Channel x" error reported from iMX6 MIPI_CSI_ERR1 register.

Maxim suggested to use MAX96705 to relace the MAX9271, it can delay the VSYNC invalid time, then the MIPI error will be fixed.

2015-11-17 update: Updated for OV10635 camera support.

File: L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2015-11-17.zip

2015-12-04 update:

File: L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2015-12-04.zip

Added patch 0009-Max9286-updated-PCLK-edge-setting-for-OV10635.patch to correct the OV10635 PCLK edge setting

2016-03-07 update:

File L3.14.38_GA_MAX9286_Surroundview_Patch_2016-03-07.zip

Added kernel patch for L3.14.38 GA 1.1.0 BSP.

2016-07-26 update:

Files: L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip; L3.14.38_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip; L3.14.52_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip.

Added gstreamer support.

Added MAX96705 support.

Added patch for L3.14.52_GA1.1.0.

2017-12-11 update:

Added CVBS surround view link: iMX6DQ TP2854 MIPI CSI2 720P CVBS camera surround view solution for Linux BSP

2021-04-26 update:

Some customer reported, when system loading is heavy, sometimes, some camera will flicker left and right. It is caused by SFMC FIFO data lost. The original patch used IDMAC 0 and IDMAC 1 for two cameras on one IPU, this is not the best setting. 

IDMAC 1 is fixed to use 1/4 SMFC FIFO and it will cause IDMAC 0 to use 1/4 SMFC FIFO too. And another 1/2 of SMFC FIFO can't be used in this case.

Some code update to improve it:

For each IPU, please use IDMAC 0 and IDMAC 2 to capture the two cameras.
This needs change the hard coding in "drivers\media\platform\mxc\capture\ipu_csi_enc.c",
"CSI_MEM1" and "IPU_IRQ_CSI1_OUT_EOF" should be changed to "CSI_MEM2" and "IPU_IRQ_CSI2_OUT_EOF". In this case, all SMFC FIFO can be used.

And in "ipu_common.c", function ipu_probe(), the followed code should be changed to make IDMAC2 use high priority too.
/* Set sync refresh channels and CSI->mem channel as high priority */
- ipu_idmac_write(ipu, 0x18800003L, IDMAC_CHA_PRI(0));
+ ipu_idmac_write(ipu, 0x1880000FL, IDMAC_CHA_PRI(0));

[dqz]

now i use BSP 3.10.53, max9286+max96707+ISX016

when connest one camera i can get image successfully,but when i connect tow cameras i always get mipi csi2 can not receive sensor clk! MIPI_CSI_PHY_STATE = 0x6f0.

i have patched L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip

my code :

 max9271_write_reg(0, 0x40, 0x2F);
 msleep(2);
 max9271_write_reg(0, 0x97, 0x5F);

....

 max9286_write_reg(0x01, 0x00);
 max9286_write_reg(0x63, 0x00);
 max9286_write_reg(0x64, 0x00);

 max9286_write_reg(0x06, 0x00);
 max9286_write_reg(0x07, 0x00);
 max9286_write_reg(0x08, 0x26);

...

mipi_csi2_reset(mipi_csi2_info, (576 * g_sensor_num) / (lanes + 1));

i have refer Debug steps for customer MIPI sensor.docx.

my doubt is how to  confirm hsp_clk  and  ccm_pixel_clkm and where to configure hsp_clk  and  ccm_pixel_clkm?

another question is about MAX9286 (FSYN CLOCKED in register 0x31 ) when i read it it is 0x0, if the two camera can not get FSYNC LOCKED, this will lead to can not receive sensor clk???
how to fix this?

thanks

[qiang_li-mpu_se]

I think your cameras are not working in sync mode. They should work in such mode: MAX9286 send a pulse to each camera, then each camera send out one frame. The VSYNC for each camera should appear in almost same time.

If MAX9286 can't sync the multi-cameras, it will not output MIPI signals. But for single camera, no sync was needed.

[dqz]

Thank you for your reply，

the camera is working in sync mode.

sometime i can connect two camrea successfully:

max9286_mipi: sensor number = 2.
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: mipi_lane_bps = 336 Mbps
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: value = 0x2a.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 1.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 2.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.

but when i try to get image,the image is not  normal,

the clock configure is not right? g_sensor_num = 2  lanes = 3

mipi_csi2_reset(mipi_csi2_info, (672* g_sensor_num) / (lanes + 1));

Qiang_FSL‌

[qiang_li-mpu_se]

I think you still need check and tuning your ISX016 camera setting, the reference code is for AP0101+AR0140 camera board.

[chenglei]

Dear Qiang,

  我正在使用s32v-evb开发板调试 MAX9286 + Onsemi AR0140+AP0101+MAX9271，您那边是否有相关调试文档(documents)或者参考代码（reference code）能够提供给我一份吗？

万分感谢。

[qiang_li-mpu_se]

Please check with your S32V contact window for that, here is just for IMX6.

[dqz]

Qiang_FSL‌  @zhang hongliang 

i want to put the four Independent camera data together in the ipu driver like one of below image , so then user can just use once DQBUF call to get all the four camera data.

Any suggestions?

or

[qiang_li-mpu_se]

You can alloc a big buffer array for V4l2 capture driver with 2560*1440 pixel buffers:

For camera 1, let its DMA buffer base address to pixel (0, 0), width 1280, height 720, but set the stride to 1280*2*2; then after filled 1280 pixels (1280*2 bytes for UYVY input), it will jump 1280*2*2 bytes; so camera 1 will fill the left-up 1280*720 pixels of the buffer;

For camera 2, let its DMA buffer base address to pixel (1280, 0), width 1280, height 720, but set the stride to 1280*2*2; then after filled 1280 pixels (1280*2 bytes for UYVY input), it will jump 1280*2*2 bytes; so camera 2 will fill the right-up 1280*720 pixels of the buffer; 

For camera 3, let its DMA buffer base address to pixel (0, 720), width 1280, height 720, but set the stride to 1280*2*2; then after filled 1280 pixels (1280*2 bytes for UYVY input), it will jump 1280*2*2 bytes; so camera 3 will fill the left-down 1280*720 pixels of the buffer; 

For camera 4, let its DMA buffer base address to pixel (1280, 720), width 1280, height 720, but set the stride to 1280*2*2; then after filled 1280 pixels (1280*2 bytes for UYVY input), it will jump 1280*2*2 bytes; so camera 4 will fill the right-down 1280*720 pixels of the buffer.

[dqz]

Thanks for your reply.

where should i change the ipu driver code?like csi_enc_setup in ipu_csi_enc.c ,and  camera_callback in mxc_v4l2_capture.c

Could you please describe in more detail?

thanks

[dqz]

Hi li

How to use gstreamer record the four channel camera data?

[qiang_li-mpu_se]

VPU performance is not enough for 4x720P30 recoder.

For one camera:

grecorder-1.0 --camera_id 0 --video_format 4 --width1280 --height 720 --fps 30

[dqz]

where can i get grecorder-1.0

[qiang_li-mpu_se]

3.14.52 BSP release has this tool. For other release, it is same as normal camera recode.

[leechanny]

Dear Qiang!

I am using MAX9286+OV10635+MAX9271. I have tested 3.10.53patch and 3.14.52patch, but both of them are not work. When I ran "/mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 1 -x 0 -g2d &". I got the same errors, like this

[ 2027.062578] max9286_mipi 0-0048: max9286_write_reg reg = 0x15, val = 0x13.
[ 2027.070238] max9286_mipi 0-0048: max9286_write_reg reg = 0xe, val = 0x5f.
root@wandboard-quad:/usr/src# [ 2027.093488] max9286_mipi 0-0048: max9286_write_reg reg = 0x3f, val = 0x4f.
[ 2027.113488] max9286_mipi 0-0048: max9286_write_reg reg = 0x3b, val = 0x1e.
[ 2027.332778] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= 4,val=43
[ 2027.552770] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= 8,val= 1
[ 2027.573473] max9286_mipi 0-0048: max9286_write_reg reg = 0x3b, val = 0x19.
[ 2027.593452] max9286_mipi 0-0048: max9286_write_reg reg = 0x12, val = 0xf3.
[ 2027.601029] max9286_mipi 0-0048: max9286_write_reg reg = 0x1, val = 0x2.
[ 2027.713713] max9286_mipi: sensor number = 1.
[ 2027.718679] max9286_mipi 0-0048: max9286_write_reg reg = 0xe, val = 0x50.
[ 2027.728809] max9286_mipi 0-0048: max9286_write_reg reg = 0xb, val = 0xe1.
[ 2027.735691] mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: mipi_lane_bps = 192 Mbps
[ 2027.744033] mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: value = 0x44.
[ 2027.751624] max9286_mipi 0-0048: max9286_write_reg reg = 0x0, val = 0xe2.
[ 2027.760490] max9286_mipi 0-0048: max9286_write_reg reg = 0xa, val = 0x22.
[ 2027.962765] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= 0,val= 4
[ 2028.182767] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= 7,val=84
[ 2028.402761] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= 1,val=90
[ 2028.602752] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= 9,val=c4
[ 2028.802767] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= a,val=c0
[ 2029.002791] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= b,val= a
[ 2029.202766] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= c,val= 4
[ 2029.223462] max9286_mipi 0-0048: max9286_write_reg reg = 0xa, val = 0x22.
[ 2029.230955] max9286_mipi 0-0048: max9286_write_reg reg = 0xa, val = 0xff.
[ 2029.432766] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= 4,val=43
[ 2029.441134] max9286 reg 0x0A = 0xff.
[ 2029.447266] max9286_mipi 0-0048: max9286_write_reg reg = 0x34, val = 0x36.
[ 2029.652766] max9286_mipi 0-0048: max9271_read_reg:read reg error: reg= 4
[ 2029.659488] max9271 reg 0x04 = 0xffffffff.
[ 2029.862766] max9286_mipi 0-0048: max9271_read_reg:read reg error: reg= 4
[ 2029.869489] max9271 reg 0x04 = 0xffffffff.
[ 2030.072766] max9286_mipi 0-0048: max9271_read_reg:read reg error: reg= 4
[ 2030.079487] max9271 reg 0x04 = 0xffffffff.
[ 2030.282776] max9286_mipi 0-0048: max9271_read_reg:read reg error: reg= 4
[ 2030.289497] max9271 reg 0x04 = 0xffffffff.
[ 2030.492792] max9286_mipi 0-0048: ov10635_read_reg:read reg error: reg=0x300a.
[ 2030.499957] max9286_mipi 0-0048: ov10635_check_device: OV10635 hasn't been found, reg[0x300a] = 0x0.
[ 2030.510196] max9286_mipi 0-0048: max9286_write_reg reg = 0x34, val = 0xb6.
[ 2030.712770] max9286_mipi 0-0048: max9271_write_reg:write reg error:reg= 4,val=83
[ 2030.823490] max9286_mipi 0-0048: max9286_write_reg reg = 0x15, val = 0x9b.
[ 2031.442763] mipi csi2 can not receive sensor clk! MIPI_CSI_PHY_STATE = 0x200.
g_in_width = 1280, g_in_height = 800.
fb_fix.id = DISP3 FG.
fb: smem_start = 0x40800000, smem_len = 0xa8c000.
fb: frame buffer size = 0x384000 bytes.
fb: g_screen_info.xres = 1280, g_screen_info.yres = 720.
fb: g_display_left = 0.
fb: g_display_top = 0.
fb: g_display_width = 960.
fb: g_display_height = 540.
start time = 1499055466 s, 40751 us
[ 2041.482836] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0
VIDIOC_DQBUF failed.

It shows that can't find OV10635 and MAX9271. If there are something wrong with I2C drivers?

I can confirm my hardware board is OK, because I am using the "MAX9286 COAX EVKIT" board.

Could you help me?

[dqz]

Hi, Li

now i try to use VPU to encode the camera data, i use MJPEG, i do not want to use other standed (H263 H264),

the the file is a too large, 100 frames will make a file about 11M.

how to reduce MJPEG file?

thanks

[dqz]

Qiang_FSL‌

now i try use h.264,

./mxc_vpu_test.out -E "-x 0 -f 0 -c 1800 -a 30 -w 1280 -h 792 -o ss.mp4 -l 0 -b 5204"

when i try to play ss.mp4 on PC i can't play it normally, like the mp4 file has no header,

how to create a mp4 file when use H.264 recoder video data?

[joewu_msi]

Hi Li,

Why do I need to set video format to 4 (UYVY) when using grecorder tool？

[xingenyu]

hello yu  ：  能否将l3.0.35的补丁给我一份，455030349@qq.com。感激不尽

[jiangdezhi]

Hi Qiang,

How to sync the camera of the MAX9296 + 4 96705 +4camera? 

We have 2 customer want to use the IMX.6 + MAX9268+ 4 MAX96705---camera. Could you suggest how to sync the camera?

Thank you very much.

[qiang_li-mpu_se]

"MAX9296 + 4 96705 +4camera" the reference patch had already covered this, the sync setting is in camera and Maxim chip side.

[qiang_li-mpu_se]

The iMX6 SabreAuto board can only supply 5V power to MAX9286 board, so if you are using long Coax cable, there will be issue. You can rework your MAX9286 board, and supply 7~12V power to it.

[tengri]

Dear Qiang Li, could you please kindly take a look at following post and give me an insight ?

Adding X11 Interlace Video Mode Support to Yocto 

Thanks in Advance

Anuradha

[braveban]

hi zhang hongliang

您好：我现在是基于 imx6-sabresd + L3.14.28调试MAX9286+AR0140+AP0101+MAX96705，运行mxc_v4l2_tvin程序出现了和您一样的错误，ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0，同时把ov5640注释相关掉，bit 19位置为0，现在出现同样的问题，并且我在内核启动的时候看到如下的报错信息，还请您帮忙看一下：

以下还有程序运行的log：

[qiang_li-mpu_se]

Hi Brave,

That error means there is no valid MIPI CSI2 signal to IMX6, from your log, it seems there is flash in your camera module, if so, the flash setting for the camera should be algined with the AP0101 setting in max9286.c, the camera should work in frame sync mode.

On Max9286, there is a lock pin, only when the four cameras are working in frame sync mode and are locked by MAX9286, it will output valid MIPI CSI2 signal to iMX6.

[zhanghongliang]

4 in 1 HD recorder resolved？

[mingjianrf0000]

你好 Qiang Li ，

按照这个帖子的补丁，我们目前MAX9286+9271四路720P输入没问题，因为需要录像，所以要使用IPU进行RGBA32/UYUY到NV12的色彩空间转换，但是一启动IPU进行色彩空间转换，视频图像就会异常，图像抖动。720*480转化下只有一个通道有问题，720P转换下会有三个通道有问题，请问是否在此方案条件下，不能再使用IPU进行色彩空间不换了，还是说底层驱动处理上有冲突所致？！如何消除？

我们使用的是IMX6D+linux3.14.28

[qiang_li-mpu_se]

你可以参考一下mxc_vpu_test的代码，camera driver可以直接把UYVY的图输出成NV12的，然后直接编码，不需要额外的转换。

另外，4路720P30的编码在VPU这边是支持不了的。

[mingjianrf0000]

谢谢你的回复，

我们的应用场景时这样的，把4路拼接后的显示画面再编码存储，所以相当于只进行一路720P的编码，由于不是原始图像的直接编码所以没办法通过camera driver把UYVY的图输出成NV12的进行编码。我们的拼接是通过GPU实现的，送给编码器的是RGBA32的，所以需要在编码前进行色彩空间转换，然后就出现了图像不正常的现象。

实际上我们也测试了只针对单通道720×480图像的色彩转化，也即先把720P的图像通过GPU缩小到720×480然后在用此分辨率的图像进行转码并录像，同样会引起输入图像异常，如果我们转码的是720×480的（RGBA32——>NV12或者UYVY——>NV12）,只有一个通道出现异常。如果转码的是1280×720的则除了第一个通道正常外，其余3个通道全部异常。

检查了底层驱动，数据流是CSI-->SMFC-->IDMAC-->MEM，跟踪IDMAC通道使用情况，默认是
    CSI_MEM0 = _MAKE_CHAN(15, NO_DMA, NO_DMA, NO_DMA, 0),
    CSI_MEM1 = _MAKE_CHAN(16, NO_DMA, NO_DMA, NO_DMA, 1),
    CSI_MEM2 = _MAKE_CHAN(17, NO_DMA, NO_DMA, NO_DMA, 2),
    CSI_MEM3 = _MAKE_CHAN(18, NO_DMA, NO_DMA, NO_DMA, 3),
720×480下出问题的是CSI_MEM1 = _MAKE_CHAN(16, NO_DMA, NO_DMA, NO_DMA, 1),如果把上述通道控制修改如下
    CSI_MEM0 = _MAKE_CHAN(15, NO_DMA, NO_DMA, NO_DMA, 1),
    CSI_MEM1 = _MAKE_CHAN(16, NO_DMA, NO_DMA, NO_DMA, 0),
    CSI_MEM2 = _MAKE_CHAN(17, NO_DMA, NO_DMA, NO_DMA, 2),
    CSI_MEM3 = _MAKE_CHAN(18, NO_DMA, NO_DMA, NO_DMA, 3),
则问题通道变成CSI_MEM0 = _MAKE_CHAN(15, NO_DMA, NO_DMA, NO_DMA, 1),从这个角度看，似乎是驱动对底层资源的管理出现冲突所致。亦或者硬件本身不支持这种用法。是否可以帮忙确认问题产生的根源，以及是否有建议的思路？

[richogrady]

Hello Qiang,

The attached tvin source code includes a local file #include "g2d.h".  However this file is not included in the archive.  Where do we get this file?

[qiang_li-mpu_se]

This file can be found from your rootfs.

[qiang_li-mpu_se]

When you used IPU for CSC on 1280*720 frames, the IPU loading is heavy, there is chance that IPU can't service the capture on time.

To improve it, you need use IDMAC 0 and IDMAC 2, then all FIFO can be used:

In ipu_common.c, function ipu_probe(), please make sure all CSI->MEM channels are set to high priority:

 /* Set sync refresh channels and CSI->mem channel as high priority */
- ipu_idmac_write(ipu, 0x18800003L, IDMAC_CHA_PRI(0));

+ ipu_idmac_write(ipu, 0x1880000FL, IDMAC_CHA_PRI(0));

[yufe]

hi, Qiang Li

my custom board is ref to saresd, max9286+max96705+ap010.

my yocto version is 3.14.52, i  got message below after patched the kernel

pr_info("max9286_mipi: sensor number = %d.\n", g_sensor_num);

if (g_sensor_num == 0) {
pr_err("%s: no camera connected.\n", __func__);
return -1;
}

can you please give me some suggestion to find the program?

[qiang_li-mpu_se]

Hi Yu fe,

This error means MAX9286 can't detect the cameras, you need check your hardware signal and power between MAX9286 and MAX96705.

[mingjianrf0000]

你好 Qiang Li ，

感谢协助，如你提供的方案，问题已经获得解决！之前所述会出现系统不稳定的问题是其它原因所致，于此方案无关！

[lemficb]

Hi Qiang,

     　Have a nice holiday. I have some trouble with  my borad. We refer to the sabresd and max9286+max96705+ap 0101, and we used the 3.14.52 BSP.  Now our four camera are all working properly, but when I used '-g2d'  in the test command  there is a error just like 'Segmentation fault'. I think It may be  something wrong with  my GPU,but I don't kown how to deal with the problem. Could you give  some advices to me ?Thanks . Here is the log below.

root@imx6qdlsolo:~# ./mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 0 -x 0
------------------------------here is AP0101--------------------------------
max9286_mipi: sensor number = 4.
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: mipi_lane_bps = 594 Mbps
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: value = 0x2e.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 1.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 2.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 3.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 4.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.
g_in_width = 1280, g_in_height = 720.
fb_fix.id = DISP4 BG.
It is background screen, only full screen default format was supported.
fb: smem_start = 0x25600000, smem_len = 0x546000.
fb: frame buffer size = 0x1c2000 bytes.
fb: g_screen_info.xres = 1280, g_screen_info.yres = 720.
fb: g_display_left = 0.
fb: g_display_top = 0.
fb: g_display_width = 1280.
fb: g_display_height = 720.
start time = 1449238585 s, 124037 us
^CERROR: v4l2 capture: mxc_v4l_dqueue() interrupt received

root@imx6qdlsolo:~# ./mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 0 -x 0 -g2d
------------------------------here is AP0101--------------------------------
max9286_mipi: sensor number = 4.
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: mipi_lane_bps = 594 Mbps
mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: value = 0x2e.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 1.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 2.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 3.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.
max9286_mipi 1-006a: ap0101_initialize: AP0101 was found, index = 4.
max9286_mipi 1-006a: ap0101_initialize:AP0101 is already in Slave (Shutter-Sync) Mode.
g_in_width = 1280, g_in_height = 720.
fb_fix.id = DISP4 BG.
fb: smem_start = 0x25600000, smem_len = 0x546000.
fb: frame buffer size = 0x1c2000 bytes.
fb: g_screen_info.xres = 1280, g_screen_info.yres = 720.
fb: g_display_left = 0.
fb: g_display_top = 0.
fb: g_display_width = 960.
fb: g_display_height = 540.
Segmentation fault

With my best regards

 Jun

[qiang_li-mpu_se]

That means your kernel is not aligned with the GPU libs in your rootfs.

[lionwei]

Hi Qiang

I am using following hardwares for similiar surroundview implementation under linux 3.14.52
ov10635 <--> ds90ub913 <-->
ov10635 <--> ds90ub913 <--> DS90ub964-q <-----> imx6q
ov10635 <--> ds90ub913 <-->
ov10635 <--> ds90ub913 <-->

the driver patches are most the same as the one you provided
and currently the mipi-csi2 can detect the pixclock successfully
also the waves on the data lines of mipi-csi2 seems correctly

however, I can not receive the camera_callback for camera 0-2, only can receive camera_callback of camera 3

I have read all the comments above, and can not find the similiar causes

I have checked the following things
1) the MIPI_IPU1_MUX and MIPI_IPU2_MUX are set to 0 for mipi_csi
2) the register dumps:
IPU1_CSI0_SENSE_CONF(0x2630000): 0x04008a00
IPU1_CSI1_SENSE_CONF(0x2638000): 0x04008a00
IPU2_CSI0_SENSE_CONF(0x2a30000): 0x04008a00
IPU2_CSI1_SENSE_CONF(0x2a38000): 0x04008a00

IPU1_IDMAC_CH_EN_1(0x2608004): 0x00000003
IPU1_IDMAC_CH_EN_2(0x2608008): 0x00000000
IPU2_IDMAC_CH_EN_1(0x2a08004): 0x00800003
IPU2_IDMAC_CH_EN_2(0x2a08008): 0x00000000

IPU1_IDMAC_CH_BUSY_1(0x2608100): 0x00000003
IPU2_IDMAC_CH_BUSY_1(0x2a08100): 0x00800002

IPU1_SMFC_MAP(0x2650000): 0x00000020
IPU2_SMFC_MAP(0x2a50000): 0x00000020

my questions are
1) are the smfc and idmac setting correct for these four cameras?
2) from the spec, it says ipu mipi should use non-gated mode, but mxc_v4l2_capture.c uses gated mode in mxc_v4l2_s_param
which mode should I use(I have tried both modes, and neither of them works for me)
3) which case need me to set the hsync_pol, vsync_pol?
4) any suggestion to identify this issue?

thanks

[qiang_li-mpu_se]

Hi Lion, no special modification is needed on iMX6 side, you should tune the Ti chip setting.

As I know, some customer had already make the Ti 964 work on iMX6 platform, the main work is in Ti side, you should check with them. For iMX6, you just need make sure the four cameras is transfered on 4 MIPI virtual channel.

[lionwei]

Hi Qiang

is there any way on imx6 to confirm the mipi data correct or not from mipi csi-2?

for example some register show the status of the mipi csi-2, specially the virtual channel/frame size related

[mustafabakircio]

Hi lionwei Qiang_FSL ;

According to iMX6 CSI2IPU documentation we need to create  short or long type packages  and also we need Start Of Transmission and End Of Transmission packages too, 

How can I send this packages for example in the camera device driver source this packages should be created and sent these data types or we are defining these values to another files . What is the format of SoT and EoT packages?

Can someone give me any example about that, I have been working to solve the csi2 format mipi clock and data lane problem, I couldnt define to the iMX6Q processor custom board since 3 week. It would be very very appreciated.

Best Regards

[qiang_li-mpu_se]

Hi Mustafa, the information can be found from MIPI CSI2 specification:

[qiang_li-mpu_se]

Hi Lion,

The MIPI CSI2 error registers can be used to check the MIPI CSI2 package is correct with CRC or follows the protocol.

[mustafabakircio]

Thank you for respond Qiang_FSL‌

I am now looking at the MIPI CSI2 Reference Documentation , I am checking all. Also I made search on the my imx6q(Wandboard Quad) custom board source and header files. After my research I could not find  relative source code in the my kernel files, which defines packet types and communication sub-structures for CSI-2 Interface Low Level Protocol message protocol , SoT and EoT package sample codes. I will take the video by using  "mxc" relative files and it's ipu source codes. So If you can share any source code or any relative codes which informs me about  SoT,EoT and LLP Massage Package Structures sample codes, It would be very nice and useful for me because I don't have any reference code in my kernel files. 

Best Regards.

[qiang_li-mpu_se]

Hi Mustafa,

These packages are implemented by the MIPI CSI2 hardware, from software side, we just tell it the virtual channel number and Data Identifier (It can be found at include\linux\mipi_csi2.h).

[lionwei]

Hi Qiang

thanks for your help!

after tune the ti settings, I can receive four camera data now.

currently for 1/2/3 sensors, I can receive the data with correct pixels

however for 4 sensors, the pixel data has some line corruptions(see the below picture)

from TI side, we can set the data lane numbers and data lane rate

however, the data lane rate has only three options(400M, 800M, 1.6G)

with different rate+lane number setting, the 1/2/3 sensors can work perfectly

for 4 sensors, only 800M+4 data lanes works, but the data has some corruptions

my question is

1) for imx6 mipi csi2 receiver, is there any special requirement on the rate and data lanes setting?

2) for imx6 mipi csi, what is the suitable ratio between the real data rate and the total data capability?

2) any suggestions?

[qiang_li-mpu_se]

Hi Lion Wei, the MAX throughput for IMX6 MIPI CSI2 is 3.2Gbps in 4 lanes mode, so each camera can be up to 800Mbps (include blank data).

And the followed code should be adjust for your real camera, the 768 should be your real camera bps:

 /* 48MHz pixel clock (1280*800@30fps) * 16 bits per pixel (YUV422) = 768Mbps mipi data rate for each camera */
 mipi_csi2_reset(mipi_csi2_info, (768 * g_sensor_num) / (lanes + 1));

Anoter limitation, at least 3 blank lines are needed for two frames from camera.

[lionwei]

Hi Qiang

I also use ov10635, and use  the same sensor register values as in your patch

the sensor output 1280*800@30fps with yuv422(uyvy,0x1e)

per my understanding, the camera data rate should be 30M*16bits=480Mbps

how to get the value of 768Mbps?

and in function mipi_csi2_reset, it just sets register MIPI_CSI_PHY_TST_CTL1,

from the name, it is just a test related register, is it useful for the mipi csi2 init?

thanks

[qiang_li-mpu_se]

Hi Lion,

You need include blank data too. So use the pixel clock to calculate the throughput is more simple and accuracy.

[zhanghongliang]

Qiang Li，我现在在调MAX96705+AP0101+AR0141，基于Linux 3.14.52版本，提示错误信息如下，请帮忙看看是哪里出了问题。

-oh 540 -d 1 -x 0 -g2da/usb/CameraTest# ./mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960
[ 34.334381] max9286_read_reg 0x49 = 1.
[ 34.338883] mipi_csi2_set_lanes = 3.
[ 34.342494] mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: mipi_lane_bps = 148 Mbps
[ 34.350399] mxc_mipi_csi2 21dc000.mipi_csi: mipi_csi2_reset: value = 0x42.
[ 34.555059] max9286_mipi 2-006a: ap0101_initialize: AP0101 was found, index = 1.
[ 34.563125] max9286_mipi 2-006a: ap0101_initialize:AP0101's 0xC88D is 0x0.
[ 34.570775] max9286_mipi 2-006a: ap0101_initialize:AP0101 is not in Host Configuration Mode. reg = 0x4.
[ 34.580281] max9286_mipi 2-006a: ap0101_initialize:AP0101's 0xdc07 is 0x4.
[ 36.453930] max9286_mipi 2-006a: ap0101_IssueHostCommand: timeout.
[ 36.460877] max9286_mipi 2-006a: ap0101_ApplyPatch: PATCH_APPLY_PATCH error.
[ 36.468012] max9286_mipi 2-006a: ap0101_LoadAp0101Patch: Patch111 load failed, result = 1794
[ 36.476664] max9286_mipi 2-006a: ap0101_initialize: LoadAp0101Patch failed, result = 1794
[ 36.613908] max9271_dump_registers: index = 1.
[ 36.618960] MAX9271 Reg 0x00 = 0x82.
[ 36.623149] MAX9271 Reg 0x01 = 0xd4.
[ 36.627402] MAX9271 Reg 0x02 = 0x1f.
[ 36.631579] MAX9271 Reg 0x03 = 0x0.
[ 36.635802] MAX9271 Reg 0x04 = 0x83.
[ 36.639915] MAX9271 Reg 0x05 = 0x0.
[ 36.643952] MAX9271 Reg 0x06 = 0xa0.
[ 36.648075] MAX9271 Reg 0x07 = 0x84.
[ 36.652172] MAX9271 Reg 0x08 = 0x0.
[ 36.656243] MAX9271 Reg 0x09 = 0x92.
[ 36.660336] MAX9271 Reg 0x0a = 0x90.
[ 36.664482] MAX9271 Reg 0x0b = 0x8a.
[ 36.668582] MAX9271 Reg 0x0c = 0x82.
[ 36.672672] MAX9271 Reg 0x0d = 0xb6.
[ 36.676826] MAX9271 Reg 0x0e = 0x2.
[ 36.680840] MAX9271 Reg 0x0f = 0x3e.
[ 36.684991] MAX9271 Reg 0x10 = 0x12.
[ 36.689069] MAX9271 Reg 0x11 = 0x0.
[ 36.693058] MAX9271 Reg 0x12 = 0x40.
[ 36.697179] MAX9271 Reg 0x13 = 0x2.
[ 36.701168] MAX9271 Reg 0x14 = 0x60.
[ 36.705282] MAX9271 Reg 0x15 = 0xe3.
[ 36.709358] MAX9271 Reg 0x16 = 0x1f.
[ 36.713434] MAX9271 Reg 0x17 = 0x99.
[ 36.717557] MAX9271 Reg 0x18 = 0xff.
[ 36.721633] MAX9271 Reg 0x19 = 0xff.
[ 36.725747] MAX9271 Reg 0x1a = 0xff.
[ 36.729823] MAX9271 Reg 0x1b = 0xff.
[ 36.733918] MAX9271 Reg 0x1c = 0x0.
[ 36.737938] MAX9271 Reg 0x1d = 0x0.
[ 36.741931] MAX9271 Reg 0x1e = 0x41.
[ 36.746045] MAX9271 Reg 0x1f = 0x1.
[ 36.749624] Dump MAX9286 registers:
[ 36.753611] MAX9286 Reg 0x00 = 0xe1.
[ 36.757734] MAX9286 Reg 0x01 = 0x0.
[ 36.761722] MAX9286 Reg 0x02 = 0x0.
[ 36.765750] MAX9286 Reg 0x03 = 0x1.
[ 36.769738] MAX9286 Reg 0x04 = 0x0.
[ 36.773727] MAX9286 Reg 0x05 = 0x0.
[ 36.777758] MAX9286 Reg 0x06 = 0x60.
[ 36.781836] MAX9286 Reg 0x07 = 0x89.
[ 36.785948] MAX9286 Reg 0x08 = 0x25.
[ 36.790025] MAX9286 Reg 0x09 = 0xd4.
[ 36.794114] MAX9286 Reg 0x0a = 0xff.
[ 36.798198] MAX9286 Reg 0x0b = 0xe4.
[ 36.802276] MAX9286 Reg 0x0c = 0x99.
[ 36.806407] MAX9286 Reg 0x0d = 0xf3.
[ 36.810484] MAX9286 Reg 0x0e = 0x50.
[ 36.814604] MAX9286 Reg 0x0f = 0xb.
[ 36.818595] MAX9286 Reg 0x10 = 0x0.
[ 36.822583] MAX9286 Reg 0x11 = 0x0.
[ 36.826610] MAX9286 Reg 0x12 = 0xf3.
[ 36.830686] MAX9286 Reg 0x13 = 0xf.
[ 36.834717] MAX9286 Reg 0x14 = 0xe4.
[ 36.838794] MAX9286 Reg 0x15 = 0x9b.
[ 36.842869] MAX9286 Reg 0x16 = 0x0.
[ 36.846895] MAX9286 Reg 0x17 = 0x1.
[ 36.850885] MAX9286 Reg 0x18 = 0x0.
[ 36.854902] MAX9286 Reg 0x19 = 0x0.
[ 36.858891] MAX9286 Reg 0x1a = 0x60.
[ 36.862967] MAX9286 Reg 0x1b = 0x0.
[ 36.867021] MAX9286 Reg 0x1c = 0xf4.
[ 36.871098] MAX9286 Reg 0x1d = 0xff.
[ 36.875218] MAX9286 Reg 0x1e = 0x40.
[ 36.879294] MAX9286 Reg 0x1f = 0x2.
[ 36.883282] MAX9286 Reg 0x20 = 0xff.
[ 36.887394] MAX9286 Reg 0x21 = 0x0.
[ 36.891383] MAX9286 Reg 0x22 = 0x0.
[ 36.895414] MAX9286 Reg 0x23 = 0x0.
[ 36.899403] MAX9286 Reg 0x24 = 0x0.
[ 36.903391] MAX9286 Reg 0x25 = 0x0.
[ 36.907715] MAX9286 Reg 0x26 = 0x0.
[ 36.911704] MAX9286 Reg 0x27 = 0xb1.
[ 36.916104] MAX9286 Reg 0x28 = 0x0.
[ 36.920094] MAX9286 Reg 0x29 = 0x0.
[ 36.924135] MAX9286 Reg 0x2a = 0x0.
[ 36.928142] MAX9286 Reg 0x2b = 0x0.
[ 36.932132] MAX9286 Reg 0x2c = 0x0.
[ 36.936164] MAX9286 Reg 0x2d = 0x0.
[ 36.940154] MAX9286 Reg 0x2e = 0x0.
[ 36.944177] MAX9286 Reg 0x2f = 0x0.
[ 36.948180] MAX9286 Reg 0x30 = 0x0.
[ 36.952182] MAX9286 Reg 0x31 = 0x40.
[ 36.956295] MAX9286 Reg 0x32 = 0x99.
[ 36.960377] MAX9286 Reg 0x33 = 0x99.
[ 36.964497] MAX9286 Reg 0x34 = 0xb6.
[ 36.968580] MAX9286 Reg 0x35 = 0x0.
[ 36.972567] MAX9286 Reg 0x36 = 0x0.
[ 36.976594] MAX9286 Reg 0x37 = 0x0.
[ 36.980582] MAX9286 Reg 0x38 = 0x0.
[ 36.984612] MAX9286 Reg 0x39 = 0x0.
[ 36.988601] MAX9286 Reg 0x3a = 0x0.
[ 36.992589] MAX9286 Reg 0x3b = 0x19.
[ 36.996699] MAX9286 Reg 0x3c = 0x24.
[ 37.000774] MAX9286 Reg 0x3d = 0x54.
[ 37.004896] MAX9286 Reg 0x3e = 0xc8.
[ 37.008971] MAX9286 Reg 0x3f = 0x4f.
[ 37.013045] MAX9286 Reg 0x40 = 0x5a.
[ 37.017183] MAX9286 Reg 0x41 = 0x10.
[ 37.021271] MAX9286 Reg 0x42 = 0x0.
[ 37.025300] MAX9286 Reg 0x43 = 0xa0.
[ 37.029384] MAX9286 Reg 0x44 = 0x0.
[ 37.033372] MAX9286 Reg 0x45 = 0x0.
[ 37.037403] MAX9286 Reg 0x46 = 0x1.
[ 37.041392] MAX9286 Reg 0x47 = 0x11.
[ 37.045507] MAX9286 Reg 0x48 = 0xc1.
[ 37.049619] MAX9286 Reg 0x49 = 0x1.
[ 37.053607] MAX9286 Reg 0x4a = 0xff.
[ 37.057754] MAX9286 Reg 0x4b = 0x0.
[ 37.061745] MAX9286 Reg 0x4c = 0x0.
[ 37.065772] MAX9286 Reg 0x4d = 0x0.
[ 37.069761] MAX9286 Reg 0x4e = 0x5.
[ 37.073750] MAX9286 Reg 0x4f = 0x0.
[ 37.077781] MAX9286 Reg 0x50 = 0x0.
[ 37.081771] MAX9286 Reg 0x51 = 0x0.
[ 37.085798] MAX9286 Reg 0x52 = 0x0.
[ 37.089789] MAX9286 Reg 0x53 = 0x0.
[ 37.093778] MAX9286 Reg 0x54 = 0x0.
[ 37.097809] MAX9286 Reg 0x55 = 0x0.
[ 37.101798] MAX9286 Reg 0x56 = 0x0.
[ 37.105824] MAX9286 Reg 0x57 = 0x0.
[ 37.109816] MAX9286 Reg 0x58 = 0x0.
[ 37.113804] MAX9286 Reg 0x59 = 0x0.
[ 37.117834] MAX9286 Reg 0x5a = 0x0.
[ 37.121824] MAX9286 Reg 0x5b = 0x60.
[ 37.125936] MAX9286 Reg 0x5c = 0x89.
[ 37.130013] MAX9286 Reg 0x5d = 0x25.
[ 37.134119] MAX9286 Reg 0x5e = 0x0.
[ 37.138127] MAX9286 Reg 0x5f = 0xf.
[ 37.142115] MAX9286 Reg 0x60 = 0x45.
[ 37.146230] MAX9286 Reg 0x61 = 0xff.
[ 37.150306] MAX9286 Reg 0x62 = 0x1f.
[ 37.154427] MAX9286 Reg 0x63 = 0x0.
[ 37.158417] MAX9286 Reg 0x64 = 0x0.
[ 37.162405] MAX9286 Reg 0x65 = 0x12.
[ 37.166519] MAX9286 Reg 0x66 = 0x96.
[ 37.170595] MAX9286 Reg 0x67 = 0x0.
[ 37.174629] MAX9286 Reg 0x68 = 0x50.
[ 37.178708] MAX9286 Reg 0x69 = 0x0.
[ 37.182941] MAX9286 Reg 0x6a = 0x0.
[ 37.186967] MAX9286 Reg 0x6b = 0x0.
[ 37.190957] MAX9286 Reg 0x6c = 0x0.
[ 37.194976] MAX9286 Reg 0x6d = 0x0.
[ 37.198965] MAX9286 Reg 0x6e = 0x0.
[ 37.202954] MAX9286 Reg 0x6f = 0x0.
[ 37.206995] MAX9286 Reg 0x70 = 0x10.
[ 37.211071] MAX9286 Reg 0x71 = 0x1.
g_in_width = 1280, g_in_height = 720.
fb_fix.id = DISP3 FG.
fb: smem_start = 0x24a00000, smem_len = 0xbdd800.
fb: frame buffer size = 0x3f4800 bytes.
fb: g_screen_info.xres = 1920, g_screen_info.yres = 1080.
fb: g_display_left = 0.
fb: g_display_top = 0.
fb: g_display_width = 960.
fb: g_display_height = 540.
start time = 1481829237 s, 120527 us
[ 47.283991] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0
VIDIOC_DQBUF failed.

[qiang_li-mpu_se]

"ap0101_initialize:AP0101 is not in Host Configuration Mode"
This log means the camera is not in I2C config mode, so we can't initialize it. You need check your camera board, if it has flash on board, then you need update the flash ROM setting with the driver's. You can also remove the flash ROM on camera board, then the MAX9286 driver can initialize it with I2C commands.