i.mx8m plus 12 bit grayscale camera iterface issue

[tdave]

Hello,

We are trying to interface a 3rd party camera sensor configured for12 bit grayscale output with resolution of 640x480. However when trying to capture the raw image through v4l2-ctl, we are not seeing expected data in the captured frame.

We made changes suggested in https://community.nxp.com/t5/i-MX-Processors/How-to-set-up-the-MIPI-CSI-2-and-ISI-to-transfer-RAW10-.... However we are still not seeing a valid camera output through v4l2-ctl.

Below are the changes that we have made

Added the below format to mxc_isi_src_formats[]

{
	.name		= "Y12 (Grey)",
	.fourcc		= V4L2_PIX_FMT_Y12,
	.depth		= { 12 },
	.memplanes	= 1,
	.colplanes	= 1,
	.align		= 1,
}
struct mxc_isi_fmt *mxc_isi_get_src_fmt(struct v4l2_subdev_format *sd_fmt)
 	sd_fmt->format.code == MEDIA_BUS_FMT_UYVY8_2X8 ||
 	sd_fmt->format.code == MEDIA_BUS_FMT_YUYV8_2X8)
 	    index = 1;
+	else if (sd_fmt->format.code == MEDIA_BUS_FMT_Y12_1X12)
+	    index = 2;
 	else
 	    index = 0;

Added below format to mxc_isi_out_formats[]

 {
	.name		= "Y12",
	.fourcc		= V4L2_PIX_FMT_Y12,
	.depth		= { 12 },
	.color		= MXC_ISI_OUT_FMT_RAW12,
	.memplanes	= 1,
	.colplanes	= 1,
	.align		= 1,
	.mbus_code	= MEDIA_BUS_FMT_Y12_1X12,
}

Set the ISI CSC in bypass mode:

	// if (is_yuv(src_fmt->fourcc) && is_rgb(dst_fmt->fourcc)) {
	// 	/* YUV2RGB */
	// 	csc = YUV2RGB;
	// 	/* YCbCr enable???  */
	// 	val |= (CHNL_IMG_CTRL_CSC_MODE_YCBCR2RGB << CHNL_IMG_CTRL_CSC_MODE_OFFSET);
	// 	val |= (CHNL_IMG_CTRL_YCBCR_MODE_ENABLE << CHNL_IMG_CTRL_YCBCR_MODE_OFFSET);
	// } else if (is_rgb(src_fmt->fourcc) && is_yuv(dst_fmt->fourcc)) {
	// 	/* RGB2YUV */
	// 	csc = RGB2YUV;
	// 	val |= (CHNL_IMG_CTRL_CSC_MODE_RGB2YCBCR << CHNL_IMG_CTRL_CSC_MODE_OFFSET);
	// } else {
	// 	/* Bypass CSC */
 		pr_info("bypass csc\n");
 		mxc_isi->cscen = 0;
 		val |= CHNL_IMG_CTRL_CSC_BYPASS_ENABLE;

Add the Y12 format to imx8-mipi-csi2-sam.c

static int csis_s_fmt(struct v4l2_subdev *sd, struct csi_sam_format *fmt)
 	case V4L2_PIX_FMT_SRGGB12:
 	    code = MEDIA_BUS_FMT_SRGGB12_1X12;
 	    break;
+	case V4L2_PIX_FMT_Y12:
+	    code = MEDIA_BUS_FMT_Y12_1X12;
+	    break;
 	default:
 		return -EINVAL;

 
v4l2-ctl command to capture the camera frame:

v4l2-ctl --verbose --device /dev/video2 --stream-mmap --stream-to=frame.raw --stream-count=1 --set-fmt-video=width=640,height=480,pixelformat='Y12 '

ISI register dump during the capture:

[   38.142066] mxc-isi 32e00000.isi: ISI CHNLC register dump, isi0
[   38.148041] mxc-isi 32e00000.isi:            CHNL_CTRL[0x00]: e0000000
[   38.154595] mxc-isi 32e00000.isi:        CHNL_IMG_CTRL[0x04]: e000001
[   38.161123] mxc-isi 32e00000.isi:    CHNL_OUT_BUF_CTRL[0x08]: 7c707
[   38.167415] mxc-isi 32e00000.isi:         CHNL_IMG_CFG[0x0c]: 1e00280
[   38.173874] mxc-isi 32e00000.isi:             CHNL_IER[0x10]: 3cfc0000
[   38.180420] mxc-isi 32e00000.isi:             CHNL_STS[0x14]: 100
[   38.186540] mxc-isi 32e00000.isi:    CHNL_SCALE_FACTOR[0x18]: 10001000
[   38.193090] mxc-isi 32e00000.isi:    CHNL_SCALE_OFFSET[0x1c]: 00
[   38.199115] mxc-isi 32e00000.isi:        CHNL_CROP_ULC[0x20]: 00
[   38.205142] mxc-isi 32e00000.isi:        CHNL_CROP_LRC[0x24]: 00
[   38.211169] mxc-isi 32e00000.isi:      CHNL_CSC_COEFF0[0x28]: 00
[   38.217198] mxc-isi 32e00000.isi:      CHNL_CSC_COEFF1[0x2c]: 00
[   38.223222] mxc-isi 32e00000.isi:      CHNL_CSC_COEFF2[0x30]: 00
[   38.229249] mxc-isi 32e00000.isi:      CHNL_CSC_COEFF3[0x34]: 00
[   38.235274] mxc-isi 32e00000.isi:      CHNL_CSC_COEFF4[0x38]: 00
[   38.241302] mxc-isi 32e00000.isi:      CHNL_CSC_COEFF5[0x3c]: 00
[   38.247325] mxc-isi 32e00000.isi:     CHNL_ROI_0_ALPHA[0x40]: 00
[   38.253351] mxc-isi 32e00000.isi:       CHNL_ROI_0_ULC[0x44]: 00
[   38.259378] mxc-isi 32e00000.isi:       CHNL_ROI_0_LRC[0x48]: 00
[   38.265412] mxc-isi 32e00000.isi:     CHNL_ROI_1_ALPHA[0x4c]: 00
[   38.271440] mxc-isi 32e00000.isi:       CHNL_ROI_1_ULC[0x50]: 00
[   38.277465] mxc-isi 32e00000.isi:       CHNL_ROI_1_LRC[0x54]: 00
[   38.283491] mxc-isi 32e00000.isi:     CHNL_ROI_2_ALPHA[0x58]: 00
[   38.289514] mxc-isi 32e00000.isi:       CHNL_ROI_2_ULC[0x5c]: 00
[   38.295541] mxc-isi 32e00000.isi:       CHNL_ROI_2_LRC[0x60]: 00
[   38.301587] mxc-isi 32e00000.isi:     CHNL_ROI_3_ALPHA[0x64]: 00
[   38.307627] mxc-isi 32e00000.isi:       CHNL_ROI_3_ULC[0x68]: 00
[   38.313655] mxc-isi 32e00000.isi:       CHNL_ROI_3_LRC[0x6c]: 00
[   38.319683] mxc-isi 32e00000.isi: CHNL_OUT_BUF1_ADDR_Y[0x70]: 84500000
[   38.326251] mxc-isi 32e00000.isi: CHNL_OUT_BUF1_ADDR_U[0x74]: 00
[   38.332288] mxc-isi 32e00000.isi: CHNL_OUT_BUF1_ADDR_V[0x78]: 00
[   38.338323] mxc-isi 32e00000.isi:   CHNL_OUT_BUF_PITCH[0x7c]: 3c0
[   38.344437] mxc-isi 32e00000.isi:     CHNL_IN_BUF_ADDR[0x80]: 00
[   38.350482] mxc-isi 32e00000.isi:    CHNL_IN_BUF_PITCH[0x84]: 00
[   38.356517] mxc-isi 32e00000.isi:     CHNL_MEM_RD_CTRL[0x88]: 00
[   38.362569] mxc-isi 32e00000.isi: CHNL_OUT_BUF2_ADDR_Y[0x8c]: 84100000
[   38.369122] mxc-isi 32e00000.isi: CHNL_OUT_BUF2_ADDR_U[0x90]: 00
[   38.375149] mxc-isi 32e00000.isi: CHNL_OUT_BUF2_ADDR_V[0x94]: 00
[   38.381177] mxc-isi 32e00000.isi:     CHNL_SCL_IMG_CFG[0x98]: 1e00280
[   38.387638] mxc-isi 32e00000.isi:       CHNL_FLOW_CTRL[0x9c]: 00

dmesg logs showing camera driver linking to mipi and isi

[    8.149343] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]
[    8.157345] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[    8.181549] mx8-img-md: created link [test_cameras 0-0030] => [mxc-mipi-csi2.0]

We would appreciate any help or suggestion to resolve this issue.

Thanks,
Tareesh

[joanxie]

since imx8mp has ISP which can support raw data directly, why do you need to use ISI? do you want to get rgb output?

[tdave]

Hello @joanxie , Thank you for your response.

We are using ISI with our other camera which uses YUV 4:2:0 format and we wanted to keep the same interface for this other camera sensor. Is the raw mode not supported through ISI ? If we have to switch to using ISP, is there any documentation/guide I can refer to?

Thanks,
Tareesh

[joanxie]

ISI can support raw data, but couldn't handle this, if you want to use ISI, it's ok, I checked your source code, I think this should be 16

.depth		= { 12 },

[tdave]

Hello @joanxie ,

I implemented the change to depth as you suggested but still don't see a valid frame.

.name		= "Y12 (Grey)",
.fourcc		= V4L2_PIX_FMT_Y12,
.depth		= { 16 },
.memplanes	= 1,
.colplanes	= 1,
.align		= 1,

.name		= "Y12",
.fourcc		= V4L2_PIX_FMT_Y12,
.depth		= { 16 },
.color		= MXC_ISI_OUT_FMT_RAW12,
.memplanes	= 1,
.colplanes	= 1,
.align		= 1,
.mbus_code	= MEDIA_BUS_FMT_Y12_1X12,

Once I tested the depth change, I also made the following changes to imx8-mipi-csi2-sam.c

static const struct csis_pix_format mipi_csis_formats[]
+{
+	.code = MEDIA_BUS_FMT_Y12_1X12,
+	.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW12,
+	.data_alignment = 16,
+},

static void disp_mix_gasket_config(struct csi_state *state)
+ case MEDIA_BUS_FMT_Y12_1X12:
+	fmt_val = GASKET_0_CTRL_DATA_TYPE_RAW12;
+	break;

static int csis_s_fmt(struct v4l2_subdev *sd, struct csi_sam_format *fmt)
+ case V4L2_PIX_FMT_Y12:
+    code = MEDIA_BUS_FMT_Y12_1X12;
+    break;

But still no luck with capturing a valid frame

Here is the v4l2-ctl output:

root@test~# v4l2-ctl --verbose --device /dev/video2 --stream-mmap --stream-to=frame.raw --stream-count=1 --set-fmt-video=width=640,height=480,pixelformat='Y12 '
VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture Multiplanar:
        Width/Height      : 640/480
        Pixel Format      : 'Y12 ' (12-bit Greyscale)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : Raw
        Transfer Function : Default
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Full Range
        Plane 0           :
           Bytes per Line : 1280
           Size Image     : 614400
                VIDIOC_REQBUFS returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      1 bytesused: 614400 ts: 33.186146 (ts-monotonic, ts-src-eof)

[tdave]

Dump of MIPI CSI2 registers:

[  452.436044] mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
[  452.441362] mxc-mipi-csi2.0:         CSIS_VERSION[0]: 0x03060301
[  452.447413] mxc-mipi-csi2.0:        CSIS_CMN_CTRL[4]: 0x00004905
[  452.453459] mxc-mipi-csi2.0:        CSIS_CLK_CTRL[8]: 0x000f0000
[  452.459509] mxc-mipi-csi2.0:          CSIS_INTMSK[10]: 0x0fffff1f
[  452.465708] mxc-mipi-csi2.0:          CSIS_INTSRC[14]: 0x00000000
[  452.471832] mxc-mipi-csi2.0:      CSIS_DPHYSTATUS[20]: 0x000000f1
[  452.477950] mxc-mipi-csi2.0:        CSIS_DPHYCTRL[24]: 0x0d800007
[  452.484064] mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_L[30]: 0x000001f4
[  452.490175] mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_H[34]: 0x00000000
[  452.496288] mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_L[38]: 0x00000000
[  452.502402] mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_H[3c]: 0x00000000
[  452.508513] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH0[40]: 0x000000b0
[  452.514627] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH1[50]: 0x000008fd
[  452.520734] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH2[60]: 0x000008fe
[  452.526843] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH3[70]: 0x000008ff
[  452.532951] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH0[44]: 0x01e00280
[  452.539061] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH1[54]: 0x80008000
[  452.545169] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH2[64]: 0x80008000
[  452.551281] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH3[74]: 0x80008000
[  452.557390] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH0[48]: 0x00000000
[  452.563507] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH1[58]: 0x00000000
[  452.569622] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH2[68]: 0x00000000
[  452.575752] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH3[78]: 0x00000000
[  452.581875] mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
[  452.587122] mxc-mipi-csi2.0:    GPR_GASKET_0_CTRL[60]: 0xffff8000
[  452.593240] mxc-mipi-csi2.0:   GPR_GASKET_0_HSIZE[64]: 0xffff8000
[  452.599352] mxc-mipi-csi2.0:   GPR_GASKET_0_VSIZE[68]: 0xffff8000

[joanxie]

你可以参考下下面两个客户的配置，我这里没有raw12的camera可以复现这个问题，首先可以看下你的camera 是什么模式，连续还是非连续的，我看这个客户depth用12也可以成功捕获图像

“https://community.nxp.com/t5/i-MX-Processors/raw12-camera-on-imx8mp-isi/m-p/1653019”

“https://community.nxp.com/t5/i-MX-Processors/imx8mp-Failed-to-obtain-sensor-raw12-data/m-p/1537266”

[tdave]

After enabling the debug logs for the MIPI interrupts, I noticed the following in the logs:

[  475.813567] mxc-mipi-csi2.0: SOT Error: 1
[  475.817582] mxc-mipi-csi2.0: Lost Frame Start Error: 1
[  475.822723] mxc-mipi-csi2.0: status: 00031000

We have the following values in the device tree:

data-lanes = <2>;
csis-hs-settle = <13>;
csis-clk-settle = <2>;

@joanxie I would appreciate if you could give some suggestion as to what might be wrong in the configuration to cause the start of frame to be lost.

Thanks.

[joanxie]

it seems that the timing issue, you can refer to the chapter 3.2.5 DPHY high-speed settle timer of enclosed file

[tdave]

I have set the value based on the document. But still don't see any valid frame output.

csis-hs-settle = <17>;
csis-clk-settle = <2>;

Thanks,
Tareesh

[joanxie]

did you set this based on your measured clock by Oscilloscope? if no, try to measure it