How to Handle High resolution Image (5MP, 2592 x 1944) in imx8m Nano?

milav_soni · ‎05-27-2022

Hello Guys,

We have an interface "AR0521" 5mp camera sensor to "MIMX8MN1CVTIZAA" microprocessor over MIPI CSI 4 lane.

I have gone through the section "13.4.3.5 High-Res Image Handling" Reference Manual of imx8m nano, in that section.

In that, we need to connect buffers (line and output) of two adjacent ISI channels, if the resolution is more than 2K.

Currently, I have registered only one ISI channel in uP.

below is a portion of imx8mn-evk.dtsi file and log output.

cameradev: camera {
                compatible = "fsl,mxc-md", "simple-bus";
                #address-cells = <2>;
                #size-cells = <2>;
                ranges;
                status = "disabled";

                isi_0: isi@0x32e20000 {
                        compatible = "fsl,imx8mn-isi";
                        reg = <0x0 0x32e20000 0x0 0x2000>;
                        power-domains = <&dispmix_pd>;
                        interrupts = <GIC_SPI 16 IRQ_TYPE_LEVEL_HIGH>;
                        interface = <2 0 2>;
                        clocks = <&clk IMX8MN_CLK_DISP_AXI>,
                                 <&clk IMX8MN_CLK_DISP_APB>,
                                 <&clk IMX8MN_CLK_DISP_AXI_ROOT>,
                                 <&clk IMX8MN_CLK_DISP_APB_ROOT>;
                        clock-names = "disp_axi", "disp_apb", "disp_axi_root", "disp_apb_root";
                        assigned-clocks = <&clk IMX8MN_CLK_DISP_AXI_ROOT>,
                                          <&clk IMX8MN_CLK_DISP_APB_ROOT>;
                        assigned-clock-rates = <500000000>, <200000000>;
                        resets = <&isi_resets>;
                        status = "disabled";

                        cap_device {
                                compatible = "imx-isi-capture";
                                status = "disabled";
                        };
                };

                mipi_csi_1: csi@32e30000 {
                        compatible = "fsl,imx8mn-mipi-csi";
                        reg = <0x0 0x32e30000 0x0 0x10000>;
                        interrupts = <GIC_SPI 17 IRQ_TYPE_LEVEL_HIGH>;
                        clock-frequency = <333000000>;
                        clocks = <&clk IMX8MN_CLK_CAMERA_PIXEL>,
                                 <&clk IMX8MN_CLK_DISP_AXI_ROOT>,
                                 <&clk IMX8MN_CLK_DISP_APB_ROOT>;
                        clock-names = "mipi_clk", "disp_axi", "disp_apb";
                        assigned-clocks = <&clk IMX8MN_CLK_CAMERA_PIXEL>;
                        assigned-clock-parents = <&clk IMX8MN_SYS_PLL2_1000M>;
                        assigned-clock-rates = <333000000>;
                        bus-width = <4>;
                        csi-gpr = <&mipi2csi_gasket>;
                        power-domains = <&mipi_pd>;
                        resets = <&mipi_csi_resets>;
                        status = "disabled";
                };

[    7.644441] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[    7.773893] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]
[    7.838157] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[ 5896.403235] isi-capture 32e20000.isi:cap_device: Post DMA cache invalidation disabled
[ 5896.411658] isi-capture 32e20000.isi:cap_device: Post DMA cache invalidation disabled
[ 5896.418344] isi-capture 32e20000.isi:cap_device: Post DMA cache invalidation disabled

if I want to register a second ISI channel what would be the right interface in imx8mn-evk.dtsi file for ISI?

isi_1: isi@0x32e20000 {
                        compatible = "fsl,imx8mn-isi";
                        reg = <0x0 0x32e20000 0x0 0x2000>;
                        power-domains = <&dispmix_pd>;
                        interrupts = <GIC_SPI 42 IRQ_TYPE_LEVEL_HIGH>;
                        interface = <? ? ?>; ------> What is the value for 5MP?
                        clocks = <&clk IMX8MN_CLK_DISP_AXI>,
                                 <&clk IMX8MN_CLK_DISP_APB>,
                                 <&clk IMX8MN_CLK_DISP_AXI_ROOT>,
                                 <&clk IMX8MN_CLK_DISP_APB_ROOT>;
                        clock-names = "disp_axi", "disp_apb", "disp_axi_root", "disp_apb_root";
                        assigned-clocks = <&clk IMX8MN_CLK_DISP_AXI_ROOT>,
                                          <&clk IMX8MN_CLK_DISP_APB_ROOT>;
                        assigned-clock-rates = <500000000>, <200000000>;
                        resets = <&isi_resets>;
                        status = "disabled";

                        cap_device {
                                compatible = "imx-isi-capture";
                                status = "disabled";
                        };
                };

According to "imx8-isi.txt ", what are input, vcx, and output, if I want to capture 5MP images using two ISI channels.

Is there any other change required to capture the high-resolution images? like in drivers of ISI and MIPI?

Please guide me on this.

Waiting for your reply.

joanxie · ‎05-31-2022

imx8mp only one mipi csi with isi, and the max resolution is 2k, and IMX8MN don't have second isi, only imx8mp has

milav_soni · ‎05-31-2022

Hello @joanxie ,

Thank you for your reply.

But What is the configuration/Mechanism for handling 5MP Data from the sensor in imx8m nano?

Is it possible to handle 5MP Data in an imx8m Nano processor (part number: MIMX8MN1CVTIZAA)?

I have also attached the CSI Register dump when 5MP run sometimes and 5MP Do not run for your reference.

Let me know.

Thank You

artsiomstaliaro · ‎07-02-2022

Yes, its possible to get 5MPix image.

You need to avoid ISI and use only MIPI_CSI block and then get RAW data directly from the sensor. I suggest using RAW8 format to be compatible with OpenCL Debayer code.

Use this example: https://www.nxp.com/docs/en/application-note/AN12060.pdf

milav_soni · ‎06-01-2022

Hello @joanxie

Can you please look out for the above post?

I got zero value in the MIPI_CSI_FRAME_COUNTER0 register when 5 MP is not run.

What is the meaning of this? is this problem related to the D-PHY of MIPI-CSI?

Can you please help me to find out the problem?

Thank you

Milav Soni

joanxie · ‎06-04-2022

what camera format do you set? yuv, rgb or raw data? you need to check if current driver supports your format or not, if no, you need add the format by yourself

milav_soni · ‎06-05-2022

Hello @joanxie ,

Thank you for your reply.

We have set RAW10 Format in the camera driver.

I have checked in "mxc_isi_channel_set_csc" function input format and output format of mxc_isi_frame.

it is BA10 format and we have also bypassed CSC in the driver.

In MIPI CSI also we have set "0x2B - RAW10" in MIPI_CSI_ISP_CONFIG0 reg.

RAW-10 Format is already configured.

Let me know if you need anything from my side.

Thank You.

joanxie · ‎06-07-2022

current isi driver couldn't support raw10 data, you need to add them in the mxc_isi_out_formats and mxc_isi_src_formats of mxc-isi-cap.c, you also need to add this format in the mxc-mipi-csi2-sam.c and mxc-isi-core.c, did you set them? and could you confirm you can get the raw data from isi correclty?

milav_soni · ‎06-08-2022

Hello @joanxie

Thank You for your reply.

Yes, I know the current ISI driver does not support RAW10 data.

I have added RAW10 Format as per below. Could you please review it?

In mxc-isi-cap.c( original , imx8-isi-cap.c) file,

struct mxc_isi_fmt mxc_isi_out_formats[] = {
	{
		.name		= "RGB565",
		.fourcc		= V4L2_PIX_FMT_RGB565,
		.depth		= { 16 },
		.color		= MXC_ISI_OUT_FMT_RGB565,
		.memplanes	= 1,
		.colplanes	= 1,
		.mbus_code  = MEDIA_BUS_FMT_RGB565_1X16,
	}, {
		.name		= "RGB24",
		.fourcc		= V4L2_PIX_FMT_RGB24,
		.depth		= { 24 },
		.color		= MXC_ISI_OUT_FMT_BGR32P,
		.memplanes	= 1,
		.colplanes	= 1,
		.mbus_code  = MEDIA_BUS_FMT_RGB888_1X24,
	}, {
		.name		= "BGR24",
		.fourcc		= V4L2_PIX_FMT_BGR24,
		.depth		= { 24 },
		.color		= MXC_ISI_OUT_FMT_RGB32P,
		.memplanes	= 1,
		.colplanes	= 1,
		.mbus_code  = MEDIA_BUS_FMT_BGR888_1X24,
	}, {
		.name		= "YUYV-16",
		.fourcc		= V4L2_PIX_FMT_YUYV,
		.depth		= { 16 },
		.color		= MXC_ISI_OUT_FMT_YUV422_1P8P,
		.memplanes	= 1,
		.colplanes	= 1,
		.mbus_code	= MEDIA_BUS_FMT_YUYV8_1X16,
	}, {
		.name		= "YUV32 (X-Y-U-V)",
		.fourcc		= V4L2_PIX_FMT_YUV32,
		.depth		= { 32 },
		.color		= MXC_ISI_OUT_FMT_YUV444_1P8,
		.memplanes	= 1,
		.colplanes	= 1,
		.mbus_code	= MEDIA_BUS_FMT_AYUV8_1X32,
	}, {
		.name		= "NV12 (YUYV)",
		.fourcc		= V4L2_PIX_FMT_NV12,
		.depth		= { 8, 8 },
		.color		= MXC_ISI_OUT_FMT_YUV420_2P8P,
		.memplanes	= 2,
		.colplanes	= 2,
		.mbus_code	= MEDIA_BUS_FMT_YUYV8_1X16,
	}, {
		.name		= "YUV444M (Y-U-V)",
		.fourcc		= V4L2_PIX_FMT_YUV444M,
		.depth		= { 8, 8, 8 },
		.color		= MXC_ISI_OUT_FMT_YUV444_3P8P,
		.memplanes	= 3,
		.colplanes	= 3,
		.mbus_code	= MEDIA_BUS_FMT_YUV8_1X24,
	}, {
		.name		= "xBGR32",
		.fourcc		= V4L2_PIX_FMT_XBGR32,
		.depth		= { 32 },
		.color		= MXC_ISI_OUT_FMT_XRGB32,
		.memplanes	= 1,
		.colplanes	= 1,
		.mbus_code	= MEDIA_BUS_FMT_RGB888_1X24,
	}, {
		.name		= "ABGR32",
		.fourcc		= V4L2_PIX_FMT_ABGR32,
		.depth		= { 32 },
		.color		= MXC_ISI_OUT_FMT_ARGB32,
		.memplanes	= 1,
		.colplanes	= 1,
		.mbus_code	= MEDIA_BUS_FMT_RGB888_1X24,
	}, { //MS added
		.name		= "BA10",
		.fourcc		= V4L2_PIX_FMT_SGRBG10,
		.depth		= { 16 },
		.color		= MXC_ISI_OUT_FMT_RAW10,
		.memplanes	= 1,
		.colplanes	= 1,
		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,		
	}
};

/*
 * Pixel link input format
 */
struct mxc_isi_fmt mxc_isi_src_formats[] = {
	{
		.name		= "RGB32",
		.fourcc		= V4L2_PIX_FMT_RGB32,
		.depth		= { 32 },
		.memplanes	= 1,
		.colplanes	= 1,
	}, {
		.name		= "YUV32 (X-Y-U-V)",
		.fourcc		= V4L2_PIX_FMT_YUV32,
		.depth		= { 32 },
		.memplanes	= 1,
		.colplanes	= 1,
	}, { //MS added
		.name		= "SGRB10",
		.fourcc		= V4L2_PIX_FMT_SGRBG10,
		.depth		= { 16 },
		.memplanes	= 1,
		.colplanes	= 1,
	}
};

In mxc-mipi-csi2-sam.c (original imx8-mipi-csi2-sam.c) file,

static const struct csis_pix_format mipi_csis_formats[] = {
	{ 
		.code = MEDIA_BUS_FMT_SGRBG10_1X10,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW10,
		.data_alignment = 16,
	}, {
		.code = MEDIA_BUS_FMT_YUYV8_2X8,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_YCBCR422_8BIT,
		.data_alignment = 16,
	}, {
		.code = MEDIA_BUS_FMT_RGB888_1X24,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RGB888,
		.data_alignment = 24,
	}, {
		.code = MEDIA_BUS_FMT_UYVY8_2X8,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_YCBCR422_8BIT,
		.data_alignment = 16,
	}, {
		.code = MEDIA_BUS_FMT_VYUY8_2X8,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_YCBCR422_8BIT,
		.data_alignment = 16,
	}, {
		.code = MEDIA_BUS_FMT_SBGGR8_1X8,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW8,
		.data_alignment = 8,
	}, {
		.code = MEDIA_BUS_FMT_SBGGR10_1X10,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW10,
		.data_alignment = 16,
	}, {
		.code = MEDIA_BUS_FMT_SGBRG10_1X10,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW10,
		.data_alignment = 16,
/*	}, {
		.code = MEDIA_BUS_FMT_SGRBG10_1X10,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW10,
		.data_alignment = 16, */
	}, {
		.code = MEDIA_BUS_FMT_SRGGB10_1X10,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW10,
		.data_alignment = 16,
	}, {
		.code = MEDIA_BUS_FMT_SBGGR12_1X12,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW12,
		.data_alignment = 16,
	}, {
		.code = MEDIA_BUS_FMT_SGBRG12_1X12,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW12,
		.data_alignment = 16,
	}, {
		.code = MEDIA_BUS_FMT_SGRBG12_1X12,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW12,
		.data_alignment = 16,
	}, {
		.code = MEDIA_BUS_FMT_SRGGB12_1X12,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW12,
		.data_alignment = 16,
	},
};

and in mxc-isi-core.c (original imx8-isi-core.c) file, i have not set format.

could you please help me how i can added RAW10 format in imx8-isi-core.c file?

----------------------------------------------------------------------------------------------------------------------------

When 5MP data arrived (10.1 MB RAW Data file) Both ISI and MIPI interrupt is called correctly.

but When 5MP data does not arrive BOTH ISI and MIPI Interrupt is not executed, and I got zero frames from MIPI D PHY because I have read the MIPI register "Frame Counter (MIPI_CSI_FRAME_COUNTER_CH0)".

and can you please help me with how I can confirm that the output from ISI is correct or not?

I have checked with Size of the file and raw data bytes as per resolution and it correctly arrives when 5MP is run.

Waiting for your reply.

Thank You

Milav Soni

joanxie · ‎06-30-2022

pls refer to this, this is tested raw10 on ov5640 successfully

--- a/drivers/media/platform/imx8/mxc-isi-core.c
+++ b/drivers/media/platform/imx8/mxc-isi-core.c
@@ -108,7 +108,10 @@ void mxc_isi_adjust_mplane_format(struct mxc_isi_fmt *fmt, u32 width, u32 height
u32 bytesperline = 0;
int i;

- pix->colorspace = V4L2_COLORSPACE_JPEG;
+ //
+ // pix->colorspace = V4L2_COLORSPACE_JPEG;
+ pix->colorspace = V4L2_COLORSPACE_RAW;
+
pix->field = V4L2_FIELD_NONE;
pix->num_planes = fmt->memplanes;
pix->pixelformat = fmt->fourcc;

How to Handle High resolution Image (5MP, 2592 x 1944) in imx8m Nano?

How to Handle High resolution Image (5MP, 2592 x 1944) in imx8m Nano?

i.MX 8M | i.MX 8M Mini | i.MX 8M Nano

Linux