iMX6 camera patch to support CSI->VDI->IC->MEM and CSI->VDI->MEM capture

[#1]

The patches are based on iMX6 L3.10.53 and 3.14.52 GA BSP.

In default linux BSP, the followed two pathes were supported in kernel driver mxc_v4l2_capture.c:
CSI->IC->MEM
CSI->MEM

After appied these patches, it can support the followed path:
CSI->VDI->IC->MEM
CSI->VDI->MEM

In this mode, the VDI de-interlace will be handled on the fly, so the whole system bandwidth will be reduced.

Limitations:
1. Since the IC can only output resolution up to 1024*1024, so this is the limation on output.
2. Only VDI motion mode 2 was supported.

mxc_v4l2_tvin.zip:
It is the test aplication,
test command for CSI->VDI->IC->MEM ("-i 2" means CSI->VDI->IC->MEM path.):
./mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 800 -oh 480 -i 2 -g2d"

test command for CSI->VDI->MEM ("-i 3" means CSI->VDI->MEM path.):
./mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 800 -oh 480 -i 3 -g2d"

[phil_martin]

Looks very interesting...

Is this patch a super set of your previous de-interlace patch?

https://community.nxp.com/docs/DOC-173003 

i.e. do I just need so use this patch, and forget about the previous example?

Also, the IC has a 1024 raster height limitation, as I understand it

- however, the CSI->VDI->MEM path should be able to support 1080i -> 1080p @ 60Hz, right?

Thanks!

Phil

[qiang_li-mpu_se]

The 60fps VDI use case is for CSI->SMFC->MEM->VDI->MEM, they are two different use case.

Due to the limitation of output resolution, it can't work for 1080 output.

[phil_martin]

Qiang Li wrote:

The 60fps VDI use case is for CSI->SMFC->MEM->VDI->MEM, they are two different use case.

Due to the limitation of output resolution, it can't work for 1080 output.

Apologies for not understanding fully, but if I want to do 1080i -> 1080p de-interlace @ 60 fps

- but this 'CSI->VDI->MEM' can support this (1080p / 60) ?

- but the older patch: https://community.nxp.com/message/856215?commentID=856215&et=watches.email.thread#comment-856215 

- does support 1080p /60?

I need to convert 1080i/60 -> 1080p/60 (-->HDMI Output)

- and I am not sure the best way to go

[qiang_li-mpu_se]

Sorry for late response, the CSI->VDI->MEM and CSI->VDI->IC->MEM path can't be used for 1920x1080 input, when output from VDI and IC, there is resolution limitation, it must be less than 1024*1024. This is hardware limitation.

But you can capture the 1080I input with CSI -> MEM path, then in capture buffer, you will get 30fps 1920x1080 frames, you can send them to HDMI display, and you can also handle them with MEM->VDI->MEM, then send them to HDMI.

[braguetan]

Qiang_FSL‌

         我打了你这个补丁用于去隔行在android6.0（内核4.1.15，打补丁没有冲突）的版本上,用命令：mxc-v4l2-tvin -ol 0 -ot 0 -ow 800 -oh 480 -i 3 -g2d进行测试，但无法抓到Camera数据（我用android的app已经可能正常预览如下图）。

错误log：

130|root@sabresd_6dq:/ # mxc-v4l2-tvin -ol 0 -ot 0 -ow 800 -oh 480 -i 3 -g2d
[ 85.821151]
[ 85.821151] In MVC: mxc_v4l_open
[ 85.825943] device name is Mxc Camera
[ 85.829894] ==>func = ioctl_g_ifparm ,line = 412
[ 85.834812] ==>func = ioctl_g_fmt_cap ,line = 544
[ 85.839632] End of mxc_v4l_open: v2f pix widthxheight 288 x 352
[ 85.845675] End of mxc_v4l_open: crop_bounds widthxheight 960 x 625
[ 85.852200] End of mxc_v4l_open: crop_defrect widthxheight 960 x 625
[ 85.858672] End of mxc_v4l_open: crop_current widthxheight 960 x 625
[ 85.865162] On Open: Input to ipu size is 960 x 625
[ 85.870207] ==>func = ioctl_s_power ,line = 450
[ 85.876333] In tp2825:ioctl_init
[ 85.880848] tp2825:ioctl_dev_init
[ 85.884360] In MVC:mxc_v4l_ioctl
[ 85.888151] In MVC: mxc_v4l_do_ioctl 80685600
[ 85.892729] case VIDIOC_QUERYCAP
[ 85.897325] In MVC:mxc_v4l_ioctl
[ 85.900573] In MVC: mxc_v4l_do_ioctl c0045627
[ 85.904937] case VIDIOC_S_INPUT
[ 85.908811] In MVC:mxc_streamoff
[ 85.912054] ====>name = CSI VDI MEM
[ 85.915677] ====>vdi_enc_select
[ 86.919626] In MVC:mxc_v4l_ioctl
[ 86.922882] In MVC: mxc_v4l_do_ioctl 80085617
[ 86.927346] case VIDIOC_G_STD
[ 86.930599] In mxc_v4l2_g_std
[ 86.933593] ==>func = ioctl_g_fmt_cap ,line = 544
[ 86.938625] ==>func = tp2825_get_std ,line = 364
[ 86.943283] In MVC:mxc_v4l_ioctl
[ 86.946604] In MVC: mxc_v4l_do_ioctl 40085618
[ 86.950985] case VIDIOC_S_STD
[ 86.954224] In mxc_v4l2_s_std ff
[ 86.957781] Setting standard to PAL ff
[ 86.961834] In MVC:mxc_v4l_ioctl
[ 86.965094] In MVC: mxc_v4l_do_ioctl c02c563a
[ 86.969781] case VIDIOC_CROPCAP
[ 86.973238] In MVC:mxc_v4l_ioctl
[ 86.976527] In MVC: mxc_v4l_do_ioctl c0cc5616
[ 86.980897] case VIDIOC_S_PARM
[ 86.984219] In mxc_v4l2_s_param
[ 86.987415] ==>func = ioctl_g_parm ,line = 469
[ 86.991875] Current capabilities are 0
[ 86.995925] Current capturemode is 0 change to 0
[ 87.000905] Current framerate is 30 change to 0
[ 87.005827] ==>func = ioctl_s_parm ,line = 511
[ 87.010284] ==>func = ioctl_g_ifparm ,line = 412
[ 87.014909] clock_curr=mclk=0
[ 87.018459] ==>func = ioctl_g_fmt_cap ,line = 544
[ 87.023182] g_fmt_cap returns widthxheight of input as 960 x 625
[ 87.029731] In MVC:mxc_v4l_ioctl
[ 87.032978] In MVC: mxc_v4l_do_ioctl c0cc5605
[ 87.037632] case VIDIOC_S_FMT
[ 87.040876] In MVC: mxc_v4l2_s_fmt
[ 87.044285] type=V4L2_BUF_TYPE_VIDEO_CAPTURE
[ 87.048870] End of mxc_v4l2_s_fmt: v2f pix widthxheight 960 x 576
[ 87.054979] End of mxc_v4l2_s_fmt: crop_bounds widthxheight 960 x 625
[ 87.061462] End of mxc_v4l2_s_fmt: crop_defrect widthxheight 960 x 625
[ 87.068048] End of mxc_v4l2_s_fmt: crop_current widthxheight 960 x 576
[ 87.074603] In MVC:mxc_v4l_ioctl
[ 87.077894] In MVC: mxc_v4l_do_ioctl c0cc5604
[ 87.082306] case VIDIOC_G_FMT
[ 87.085578] In MVC: mxc_v4l2_g_fmt type=1
[ 87.089600] type is V4L2_BUF_TYPE_VIDEO_CAPTURE
[ 87.094398] End of mxc_v4l2_g_fmt: v2f pix widthxheight 960 x 576
[ 87.100536] End of mxc_v4l2_g_fmt: crop_bounds widthxheight 960 x 625
[ 87.107033] End of mxc_v4l2_g_fmt: crop_defrect widthxheight 960 x 625
[ 87.113572] End of mxc_v4l2_g_fmt: crop_current widthxheight 960 x 576
g_in_width = 960, g_in_height = 5[ 87.120298] In MVC:mxc_v4l_ioctl
76.
[ 87.126421] In MVC: mxc_v4l_do_ioctl c0145608
[ 87.131258] case VIDIOC_REQBUFS
[ 87.134665] In MVC:mxc_streamoff
fb_fix.id = DISP3 FG.
fb: smem_start = 0x2f700000, smem_len = 0x708000.
fb: frame buffer size = 0x258000 bytes.
fb: g_screen_info.xres = 1024, g_screen_info.yres = 600.
fb: g_display_left = 0.
fb: g_display_top = 0.
fb: g_display_width = 800.
fb: g_display_height = 480.
[ 87.204895] In MVC:mxc_v4l_ioctl
[ 87.208181] In MVC: mxc_v4l_do_ioctl c0445609
[ 87.212550] case VIDIOC_QUERYBUF
[ 87.216073] In MVC:mxc_v4l2_release_bufs
[ 87.220005] In MVC:mxc_v4l2_prepare_bufs
[ 87.223940] In MVC:mxc_v4l_ioctl
[ 87.227236] In MVC: mxc_v4l_do_ioctl c0445609
[ 87.231604] case VIDIOC_QUERYBUF
[ 87.235099] In MVC:mxc_v4l2_release_bufs
[ 87.239056] In MVC:mxc_v4l2_prepare_bufs
[ 87.243038] In MVC:mxc_v4l_ioctl
[ 87.246321] In MVC: mxc_v4l_do_ioctl c0445609
[ 87.251968] case VIDIOC_QUERYBUF
[ 87.255558] In MVC:mxc_v4l2_release_bufs
[ 87.259497] In MVC:mxc_v4l2_prepare_bufs
[ 87.263486] In MVC:mxc_v4l_ioctl
[ 87.268051] In MVC: mxc_v4l_do_ioctl c0445609
[ 87.273459] case VIDIOC_QUERYBUF
[ 87.277378] In MVC:mxc_v4l2_release_bufs
[ 87.281648] In MVC:mxc_v4l2_prepare_bufs
[ 87.285934] In MVC:mxc_v4l_ioctl
[ 87.290447] In MVC: mxc_v4l_do_ioctl c044560f
[ 87.295390] case VIDIOC_QBUF
[ 87.298957] In MVC:mxc_v4l_ioctl
[ 87.303373] In MVC: mxc_v4l_do_ioctl c044560f
[ 87.308147] case VIDIOC_QBUF
[ 87.312488] In MVC:mxc_v4l_ioctl
[ 87.316074] In MVC: mxc_v4l_do_ioctl c044560f
[ 87.320623] case VIDIOC_QBUF
[ 87.323940] In MVC:mxc_v4l_ioctl
[ 87.327458] In MVC: mxc_v4l_do_ioctl c044560f
[ 87.332009] case VIDIOC_QBUF
[ 87.335942] In MVC:mxc_v4l_ioctl
[ 87.339210] In MVC: mxc_v4l_do_ioctl 40045612
[ 87.343584] case VIDIOC_STREAMON
[ 87.347164] In MVC:mxc_streamon
[ 87.350326] IPU:In vdi_enc_enabling_tasks
[ 87.360613] In vdi_enc_setup
start time = 82 s, 650072 us
[ 87.363647] In MVC:mxc_v4l_ioctl
[ 87.369516] In MVC: mxc_v4l_do_ioctl c0445611
[ 87.373883] case VIDIOC_DQBUF
[ 87.377166] In MVC:mxc_v4l_dqueue
[ 97.375336] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0
VIDIOC_DQBUF failed.
[ 97.381873] In MVC:mxc_v4l_ioctl
[ 97.387275] In MVC: mxc_v4l_do_ioctl 40045613
[ 97.391845] case VIDIOC_STREAMOFF
[ 97.395911] In MVC:mxc_streamoff
[ 97.895675] In MVC:mxc_free_frames
[ 97.936805] In MVC:mxc_v4l_close
[ 97.941207] In MVC:mxc_streamoff
[ 97.944724] ==>func = ioctl_s_power ,line = 450
[ 97.949348] mxc_v4l_close: release resource
[ 97.953775] MVC: In mxc_free_frame_buf
[ 97.957621] In MVC:mxc_free_frames
root@sabresd_6dq:/ #

[nguyenmr]

I used this patch for android6, kernel 4.1.15 and have same problem, can't use path CSI->VDI->MEM. 

can this patch use for 4.1.15 or not?

[qiang_li-mpu_se]

Hi Nguyen, your resolution is strange, I think you need double check the active data size setting in VDI, if the data size is wrong, VDI can't get enough data from CSI, it will not output.

[leonlin1]

Hi Qiang,

              Can this patch apply with ISL79985 and ISL79987, L4.1.15_ISL7998x_Surroundview_Patch_20161122.zip? Becasue ISL79985 only support 720x240, the system need more overhead costs for de-interlace. Can this patch help reduce these overheads?

[crl6crl6]

Hi Qiang,

I  run mxc-v4l2-tvin  -ow 1280 -oh 720 -i 3  ,only get one frame and then get follow information ：

<4>[   72.528933] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_9 = 0x04000000

VIDIOC_DQBUF failed.

<3>[   82.525936] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

<3>[   83.087057] imx-ipuv3 2400000.ipu: warning: disable ipu dma channel 5 during its busy stat

.................................................

[stankong]

这个补丁遗漏一部分，没有在 ipu_common.c 文件中的 ipu_enable_channel 函数里添加CSI_VDI_MEM和CSI_VDI_PRP_VF_MEM 对应的enable代码。下面是补丁

From 67d877007ed11c4f897fb539d19f28e9c7505c0a Mon Sep 17 00:00:00 2001
From: stan <klshu36@163.com>
Date: Thu, 8 Feb 2018 15:50:12 +0800
Subject: [PATCH] add-CSI_VDI_MEM-to-ipu_enable_channel

---
drivers/mxc/ipu3/ipu_common.c | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/drivers/mxc/ipu3/ipu_common.c b/drivers/mxc/ipu3/ipu_common.c
index b856a35..46c34d2 100644
--- a/drivers/mxc/ipu3/ipu_common.c
+++ b/drivers/mxc/ipu3/ipu_common.c
@@ -2565,6 +2565,16 @@ int32_t ipu_enable_channel(struct ipu_soc *ipu, ipu_channel_t channel)
if (ipu->ic_use_count > 0)
ipu_conf |= IPU_CONF_IC_EN;
break;
+ case CSI_VDI_MEM:
+ case CSI_VDI_PRP_VF_MEM:
+ if (ipu->vdi_use_count > 0) {
+ ipu_conf |= IPU_CONF_ISP_EN;
+ ipu_conf |= IPU_CONF_VDI_EN;
+ ipu_conf |= IPU_CONF_IC_INPUT;
+ }
+ if (ipu->ic_use_count > 0)
+ ipu_conf |= IPU_CONF_IC_EN;
+ break;
default:
break;
}
--
1.9.1

我检查了linux代码，在linux内核版本 L3.10.53 and L3.14.52中，ipu_enable_channel 函数没有这个问题，而在L4.1.15中这个函数修改了，才出现了这个问题。所以上述补丁是基于L4.1.15。

[crl6crl6]

thank you very much。

[liyihong]

Qiang_FSL‌  how can I use this patch to L3.0.35 ?  I found the libcamera2  in android BSP use the  following setting  in DeviceAdapter.cpp :

int input = 1;
ret = ioctl(mCameraHandle, VIDIOC_S_INPUT, &input);

Do I need to modify the  int input = 1;  to  int input = 3;   ?  how to improve the android vdi ?

[liyihong]

hi brague

      你好，想问一下你测试用的是模拟摄像头吗？可以了解下你摄像头的分辨率是多少，具体显示的分辨率是多少？打了这个补丁后用app打开摄像头时显示的效果有所改善吗？你也是通过libcamera2这个hal层来显示摄像头的图像吗？我看到测试的命令“mxc-v4l2-tvin -ol 0 -ot 0 -ow 800 -oh 480 -i 3 -g2d”，i是设置成3的，而libcamera2上面是设置成1，那是否有必要修改libcamera2这个hal 层的代码？谢谢。

[qiang_li-mpu_se]

Yes, you can set input=3, for input=1, it means CSI->SMFC->MEM, it captured the original video data to memory.

[braguetan]

我们的内核是4.1.15，你先用测试用例测试，如果效果是可行的，再把相应的设置参数改到HAL层就行了。