raw12 camera on imx8mp isi

[zhangjunsheng]

Hi NXP FAEs:

    I encounter problem when I try to make camera which output raw12 data with imx8mp isi platfrom.

    the yocto branch is kirkstone and the kernel tag is: lf-5.15.y

    Please find the patch as attached I worked out for debugging the functions.

    First, I force to use mipi_csis_formats[0] in find_csis_format in imx8-mipi-csi2-sam.c, then csi/gasket register will be configured according to MIPI_CSIS_ISPCFG_FMT_YCBCR422_8BIT. when I run 'v4l2-ctl -d /dev/video2 --set-fmt-video=width=3840,height=2160,pixelformat=BG12 --stream-mmap', both csi interrupts and isi interrupts are tirggered well and there is no error such as ecc/crc reported in csi interrupts. application runs well, isi interrupt reports status 0x60020100 (bit29 shows frames are correctly received and trigger the interrupt, bit 17 reports vsync is ealier than experted (as camera outpus raw12 data and isi expecting yuv data, so frame length is not the same).

    When csi/gasket is configured according to MIPI_CSIS_ISPCFG_FMT_RAW12, application is stuck with below findings:

     1. gasket register report some pixels sent out to ISI (0x32ec007c: 0x0000FB40 and does not increase). it should increase in normal case.

     2. gasket register report no line sent out to ISI (0x32ec0080: 00000000 and does not increase). it should increase in normal case.

     3. csi register report one frame sent out to ISI (0x32e40100: 00000001 and does not increase). it should increase in normal case.

     4. csi register report no dphy error (0x32e40020: 00000000).

     5. csi register report no csi error but data overflow (0x32e40014: 00000010). looks ISI did not handle data from csi and csi data is stuck to sent to ISI.

     6. isi register report no interrupt is triggered (code is added to print something when isi interrupt is triggered, but no infomration printed).

Below is the registers dumped with memtool during runing 'v4l2-ctl -d /dev/video2 --set-fmt-video=width=3840,height=2160,pixelformat=BG12 --stream-mmap &' 

$ /unit_tests/memtool 0x32e00000 0x28
chnl_ctrl[0x00]                 e2000000    bit[25]: 2 line buffers chained (supports 4096 horizontal resolution).
chnl_img_ctrl[0x04]             0f000001    [29:24] 001111b - raw16 - 16-bit raw data packed into 32-bit dword，认为进来的数据是raw16
chnl_out_buf_ctrl[0x08]         0007c707    [15:14] load buffer 2/1 address from chnlout_buf1_addr_* registers (正常时此值来回跳动)
chnl_img_cfg[0x0c]              08700f00    (2160x3840)
chnl_ier[0x10]                  3cfc0000
chnl_sts[0x14]                  40000100    bit[9:8]: current frame being stored in buffer 2/1 address (正常时此值来回跳动) bit[30] new line received
chnl_scale_factor[0x18]         10001000
chnl_scale_offset[0x1c]         00000000
chnl_crop_ulc[0x20]             00000000
chnl_crop_lrc[0x24]             00000000
chnl_csc_coeff0[0x28]           00000000
chnl_csc_coeff1[0x2c]           00000000
chnl_csc_coeff2[0x30]           00000000
chnl_csc_coeff3[0x34]           00000000
chnl_csc_coeff4[0x38]           00000000
chnl_csc_coeff5[0x3c]           00000000
chnl_roi_0_alpha[0x40]          00000000
chnl_roi_0_ulc[0x44]            00000000
chnl_roi_0_lrc[0x48]            00000000
chnl_roi_1_alpha[0x4c]          00000000
chnl_roi_1_ulc[0x50]            00000000
chnl_roi_1_lrc[0x54]            00000000
chnl_roi_2_alpha[0x58]          00000000
chnl_roi_2_ulc[0x5c]            00000000
chnl_roi_2_lrc[0x60]            00000000
chnl_roi_3_alpha[0x64]          00000000
chnl_roi_3_ulc[0x68]            00000000
chnl_roi_3_lrc[0x6c]            00000000
chnl_out_buf1_addr_y[0x70]      d7f00000    每次可能会变化
chnl_out_buf1_addr_u[0x74]      00000000
chnl_out_buf1_addr_v[0x78]      00000000
chnl_out_buf_pitch[0x7c]        00001e00    (7680)  = width(3840) * depth(16) / 8. depth设定很重要，16位对齐
chnl_in_buf_addr[0x80]          00000000
chnl_in_buf_pitch[0x84]         00000000
chnl_mem_rd_ctrl[0x88]          00000000
chnl_out_buf2_addr_y[0x8c]      d6f00000    每次可能会变化
chnl_out_buf2_addr_u[0x90]      00000000
chnl_out_buf2_addr_v[0x94]      00000000
chnl_scl_img_cfg[0x98]          04380780
chnl_flow_ctrl[0x9c]            00000000

$ /unit_tests/memtool 0x32e40000 0x20
register                        yuyv        raw12
csis_version[0]                 03060301    03060301
csis_cmn_ctrl[4]                00004b05    00004b05
csis_clk_ctrl[8]                000f0000    000f0000    bit[4] 0-pixel clock is always alive, 1-pixel clock is alive during the interval of frame
csis_intmsk[10]                 0fffff1f    0fffff1f
csis_intsrc[14]                 00000000    00000010    正常情况为0，偶尔也能读到错误
csis_dphystatus[20]             000000f0    000000f0    bit[7-4]: data lane [3:0] is in stop state      
csis_dphyctrl[24]               1300001f    1300001f    bit[23:22] - clksettle. 2'b0x - 110ns to 280ns (v0.87 to v1.00)  bit[31:24]: hssettle
csis_dphybctrl_l[30]            000001f4    000001f4
csis_dphybctrl_h[34]            00000000    00000000
csis_dphysctrl_l[38]            00000000    00000000
csis_dphysctrl_h[3c]            00000000    00000000
csis_ispconfig_ch0[40]          00001078    000020B0    不同
csis_ispresol_ch0[44]           08700f00    08700f00    (2160x3840)
csis_ispsync_ch0[48]            00000000    00000000    bit[23-18]: Interval between last falling of DVALID and falling of HSync (cycle of Pixel clock)
csis_ispconfig_ch1[50]          000008fd    000008fd
csis_ispresol_ch1[54]           80008000    80008000
csis_ispsync_ch1[58]            00000000    00000000
csis_ispconfig_ch2[60]          000008fe    000008fe
csis_ispsync_ch2[68]            00000000    00000000
csis_ispresol_ch2[64]           80008000    80008000
csis_ispconfig_ch3[70]          000008ff    000008ff
csis_ispresol_ch3[74]           80008000    80008000
csis_ispsync_ch3[78]            00000000    00000000

$ /unit_tests/memtool 0x32ec0060 3
register                    yuyv        raw12
GPR_GASKET_0_CTRL[60]       00001E03    00002C01    bit[15:8] 1E: yuyv, 2C: raw12
GPR_GASKET_0_HSIZE[64]      00000F00    00000F00    3840
GPR_GASKET_0_VSIZE[68]      00000870    00000870    2160

   I checked many threads in nxp community and did not find the solution.

  Please please to advise how to cofigure csi/gasket/isp registers so I can make raw12 data output on imx8mp isi system.

Best regards.
Johnson

[joanxie]

pls share your dts file and MIPI_CSIx_DPHY_STATUS with me, what camera do you use?

[zhangjunsheng]

Hi @joanxie 

   the camera is imx715.

   dts as below

&isi_0 {
	status = "okay";

	cap_device {
		status = "okay";
	};
};

&mipi_csi_0 {
	status = "okay";

	port@0 {
		mipi_csi0_ep: endpoint {
			remote-endpoint = <&imx715_mipi_0_ep>;
			data-lanes = <4>;
			csis-hs-settle = <19>;
			csis-wclk;
		};
	};
};

&iomuxc {
	pinctrl_csi0_rst: csi0_rst_grp {
		fsl,pins = <
			MX8MP_IOMUXC_ECSPI1_SCLK__GPIO5_IO06	0x19
		>;
	};
};

&i2c3 {
	imx715_csi1: imx715_mipi@1a {
		compatible = "leopardimage,imx715";
		reg = <0x1a>;
		pinctrl-names = "default";
		pinctrl-0 = <&pinctrl_csi0_rst>;
		reset-gpios = <&gpio5 6 GPIO_ACTIVE_HIGH>;
		status = "okay";

		port {
			imx715_mipi_0_ep: endpoint {
				data-lanes = <1 2 3 4>;
				clock-lanes = <0>;
				remote-endpoint = <&mipi_csi0_ep>;
			};
		};
	};
};

    dphy status is 0x0 when application is stuck.

    I also compared the csi/gasket registers with imx8mp-evk board during basler camera capturing on imx8mp-evk board, it's the same.

    so are wondering if csi format is configured to raw12, only can cowork with ISP, and can not cowork with ISI?

Best regards.

Johnson

[joanxie]

ISI can support raw data, so the current issue is camera sent raw12 data, but you get the data is still yuv?could you read the data from ISI? data should be stored as little endian in ISI, and pls share your patch for dts file with me

[zhangjunsheng]

Hi @joanxie 

    No, my issue is when CSI is set as RAW12, there is no data in ISI (application is stuck and there is no isi interrupt monitored).

    Actually camera send out RAW12 format of data.

    For debugging, I have ever set CSI format as YUYV (MEDIA_BUS_FMT_UYVY8_2X8 in imx8-mipi-csi2-sam.c), there is data output from ISI, application running well, but the data are not correct).

    So, My questions is when CSI format is set as RAW12, why there is no interrupt in ISI?

    You can read the registers I dumped to understand what I configured and point out which register I configured wrong and how I should configure to make CSI/ISI work for RAW12.

Best regards.

Johnson

[joanxie]

could you share logfile and dts file with me? did you measure the clock from mipi csi, is it ok?

[zhangjunsheng]

Hi @joanxie 

     I shared the dts already and have shared the dumped registers.

     Sorry I don't have the high frequency osc to measure the mipi signals. but from the interrupt triggering message printed, near no ecc/crc error.

     below is the log when set output format as RGB3 (just for debugging, actually it should be BG12). each interrupt bit is printed no more than 5 times to avoid huge message dumped.

root@vest-e2i-plus:~# v4l2-ctl -d /dev/video2 --set-fmt-video=width=3840,height=2160,pixelformat=RGB3 --stream-mmap --stream-count 1 --stream-to raw
[   20.308872] bypass csc
[   20.311256] input fmt RGB4                 V4L2_PIX_FMT_RGB32 ()
[   20.313995] output fmt RGB3                V4L2_PIX_FMT_RGB24/MEDIA_BUS_FMT_RGB888_1X24 (MXC_ISI_OUT_FMT_BGR32P = 0x07 BGR888P - 8-bits per color component (packed into 24-bits).
[   20.961709] csi interrupt bit: 2 0         page 5374: ECC error
[   20.965126] csi interrupt bit: 17 0        page 5374: Start of transmission error, [Lane3,Lane2,Lane1,Lane0]
[   20.968613] csi interrupt bit: 19 0
[   20.972126] csi interrupt bit: 12 0        page 5374: Indication of lost of Frame Start packet, CH0.
[   20.975633] csi interrupt bit: 12 1
[   20.979136] csi interrupt bit: 12 2
[   20.982638] csi interrupt bit: 12 3
[   20.986138] csi interrupt bit: 20 0        page 5374: FE packet is received, CH0.
[   20.989626] csi interrupt bit: 24 0        page 5374: FS packet is received, CH0.
[   21.018115] csi interrupt bit: 20 1
[   21.021614] isi interrupt bit: 9 0         page 5304: Current frame being stored in Buffer 2 Address
[   21.025017] isi interrupt bit: 17 0        page 5304: VSYNC timing (Early) error interrupt flag
[   21.028503] isi interrupt bit: 29 0        page 5303: Frame stored successfully interrupt flag
[   21.031989] isi interrupt bit: 30 0        page 5303: Line received and stored interrupt flag
[   21.035495] csi interrupt bit: 24 1
[   21.051446] csi interrupt bit: 20 2
[   21.054940] isi interrupt bit: 8 0         page 5304: Current frame being stored in Buffer 1 Address
[   21.058340] isi interrupt bit: 17 1
[   21.061827] isi interrupt bit: 29 1
[   21.065314] isi interrupt bit: 30 1
[   21.068820] csi interrupt bit: 24 2
[   21.084781] csi interrupt bit: 20 3
[   21.088287] isi interrupt bit: 9 1
[   21.091689] isi interrupt bit: 17 2
[   21.095176] isi interrupt bit: 29 2
[   21.098663] isi interrupt bit: 30 2
[   21.102174] csi interrupt bit: 24 3
[   21.118114] csi interrupt bit: 20 4
[   21.121622] isi interrupt bit: 8 1
[   21.125024] isi interrupt bit: 17 3
[   21.128511] isi interrupt bit: 29 3
[   21.131997] isi interrupt bit: 30 3
[   21.135508] csi interrupt bit: 24 4
[   21.151452] isi interrupt bit: 9 2
[   21.154870] isi interrupt bit: 17 4
[   21.158357] isi interrupt bit: 29 4
[   21.161844] isi interrupt bit: 30 4
[   21.184785] isi interrupt bit: 8 2
[   21.218118] isi interrupt bit: 9 3
[   21.251451] isi interrupt bit: 8 3
[   21.284783] isi interrupt bit: 9 4
[   21.318118] isi interrupt bit: 8 4

   when output is set to BG12, application is stuck and no isi interrupt is printed. csi interrupt report data overflow.

root@vest-e2i-plus:~# v4l2-ctl -d /dev/video2 --set-fmt-video=width=1920,height=1080,pixelformat=BG12 --stream-mmap --stream-count 1 --stream-to raw
[   32.377013] bypass csc
[   32.379387] input fmt RGB4
[   32.382140] output fmt BG12
[   33.028331] csi interrupt bit: 12 0
[   33.031854] csi interrupt bit: 12 1
[   33.035357] csi interrupt bit: 12 2
[   33.038876] csi interrupt bit: 12 3
[   33.042386] csi interrupt bit: 12 4
[   33.051416] csi interrupt bit: 20 0
[   33.054921] csi interrupt bit: 24 0
[   33.084034] csi interrupt bit: 4 0               page 5374: Overflow is caused in image FIFO
[   33.087454] csi interrupt bit: 4 1
[   33.090875] csi interrupt bit: 4 2
[   33.094291] csi interrupt bit: 4 3
[   33.097706] csi interrupt bit: 4 4

Best regards.

Johnson

[joanxie]

the reason I ask clock is that customer needs change hs-settle in the dts according to their own sensor clock if they porting new sensor, if the clock isn't correct, maybe would cause such issue, so I need to check this firstly, if you couldn't measure it, maybe you can try to change the value to see what happen

[zhangjunsheng]

Hi @joanxie :

    Sorry for late reponse as I missed your latest update. From the csi interrupt triggerring information dumped, I did not intend to think the problem is on the mipi csi signal side as no crc or ecc error reported when hang, so I think it may be caused by ISI side, and then try to make camera work in pipe CSI0 - ISP0, but unluckily when using ISP pipe, I face the same issue.

    So I go back to check the issue in ISI pipe.

    Folowing your suggestion, I changed the value of "csis-hs-settle" in dts as each integer from 33 to 7, and also try with/without "csis-wclk" for each value, with below finding:

    1. when "csis-hs-settle" is set in the range from 33 to 9, the csi interrput triggering information dumped is the same, capture application is stuck.

    2. when "csis-hs-settle" is set in the range from 8 to 7, more intrrupt bit 16/17/19 is triggered, capture application is stuck.

    3. switch between with/without "csis-wclk", no affect.

&mipi_csi_0 {
    status = "okay";

    port@0 {
        mipi_csi0_ep: endpoint {
            remote-endpoint = <&imx715_mipi_0_ep>;
            data-lanes = <4>;
            csis-hs-settle = <33>;
            csis-wclk;
        };
    };
};

    the camera mipi speed is 891M per lane, which "csis-hs-settle" value is the best?

    For your information I have ever changed the CSI and ISI pixel clock to 1000M (currently is 500M), no help.

     And the most interesting point is:

drivers/staging/media/imx/imx8-mipi-csi2-sam.c

drivers/staging/media/imx/imx8-mipi-csi2-sam.c

static const struct csis_pix_format mipi_csis_formats[] = {
    {
        .code = MEDIA_BUS_FMT_YUYV8_2X8,
        .fmt_reg = MIPI_CSIS_ISPCFG_FMT_YCBCR422_8BIT,
        .data_alignment = 16,
    },

1. camera output data format is RAW12.

2. when fmt_reg is MIPI_CSIS_ISPCFG_FMT_YCBCR422_8BIT, data_alignment is 16, CSI can recevie data, capture application did not hang (in that case data application recevied is not correct, csi interrupt does not report error).

3. when fmt_reg is MIPI_CSIS_ISPCFG_FMT_RAW12, data_alignment is 32, CSI can recevie data, capture application did not hang (in that case data application recevied is not correct, , csi interrupt does not report error).

4. when fmt_reg is MIPI_CSIS_ISPCFG_FMT_RAW12, data_alignment is 16, CSI reports image data overflow, capture application hang (in that case csi still does not report error such as crc/ecc error).

I can not understand why CSI works like that, and can not find the answer from imx8mp user manual, and can not find the solution from imx8mp user manual.

Besides mipi csi signal error (I don't intend to think its mipi csi signal issue as no related csi error reported), any possible cause for that?

Best regards.

Johnson

[joanxie]

could you tell me what camera do you use? let me try to find more information about your camera if I could

[zhangjunsheng · Accepted solution]

Hi @joanxie:

     After try, I fixed the hang issue. the cause is the mipi frame resolution camera sent out is (1944 x 1118), but the effective pixel resolution is (1920 x 1080). in mipi frames, there are also embedded data, OB data etc included. so, when I set the camera solution into (1944 x 1118) instead of (1920 x 1080), anything is ok now.

     Actually did not understand the CSI behavior I present in previous updates. but it's not important now and I will treat it as a lessen: when configured resolution is smaller than mipi frame resolution, CSI could be overflowed.

Best regards.

Johnson

[FvdWalt]

Hi @zhangjunsheng,

Thank you for this insight. We actually had the same issue where we were trying to read 12-bit data from an image sensor configured for 2x2 pixel binning. We had FIFO overflow until we increased the resolution in our ISP driver. 

Regards,