IMX290 mono camera via 4 lane MIPI CSI-2 and ISI to transfer Y10_1X10 images on the i.MX8QM

[Peter_ManyPebbles]

Hi,

I am using a Variscite SPEAR-MX8 Evaluation Kit based on a i.MX8QM processor and have created a custom MIPI CSI-2 breakout board that connects an IMX290 via 4 lanes on both MIPI-CSI2 0 and MIPI-CSI2 1 channels i.e. I am attaching two cameras.

I have ported the I2C driver for the IMX290 from here (and I believe that it is working correctly):

https://github.com/raspberrypi/linux/blob/rpi-5.15.y/drivers/media/i2c/imx290.c

I have also followed the process detailed in this post which in turn references previous work:

https://community.nxp.com/t5/i-MX-Processors/How-to-set-up-the-MIPI-CSI-2-and-ISI-to-transfer-RAW10-...

The key difference to my requirements is that the solution above is used to transfer V4L2_PIX_FMT_SGRBG10 format images whereas I am looking to transfer V4L2_PIX_FMT_Y10 (or even V4L2_PIX_FMT_Y12) images that are greyscale rather than colour.

By using the following command I am able to capture an image from the camera but the images are arriving incorrectly, containing only 64 unique values in the data (6 bit depth???):

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat='Y10 ' --stream-mmap 2 --stream-count=1 --stream-to=test.raw --verbose

My relevant Device Tree settings are as follows:

&i2c_mipi_csi0 {
/delete-node/ov5640_mipi@3c;

imx290_mipi0: imx290_mipi@1a {
compatible = "sony,imx290-mono";
reg = <0x1a>;

pinctrl-names = "default";
pinctrl-0 = <&pinctrl_mipi_csi0>;
clocks = <&xtal24m>;
clock-names = "xclk";
clock-frequency = <37125000>;
csi_id = <0>;
powerdown-gpios = <&lsio_gpio1 28 GPIO_ACTIVE_HIGH>;
reset-gpios = <&lsio_gpio1 27 GPIO_ACTIVE_LOW>;
mclk = <24000000>;
mclk_source = <0>;
mipi_csi;

status = "okay";
port {
imx290_mipi_0_ep: endpoint {
remote-endpoint = <&mipi_csi0_ep>;
data-lanes = <1 2 3 4>;
link-frequencies = /bits/ 64 <148500000 222750000 297000000 445500000>;
clocks-lanes = <0>;
};
};
};
};

&i2c_mipi_csi1 {
/delete-node/ov5640_mipi@3c;

imx290_mipi1: imx290_mipi@1a {
compatible = "sony,imx290-mono";
reg = <0x1a>;

pinctrl-names = "default";
pinctrl-0 = <&pinctrl_mipi_csi1>;
clocks = <&xtal24m>;
clock-names = "xclk";
clock-frequency = <37125000>;
csi_id = <1>;
powerdown-gpios = <&lsio_gpio1 31 GPIO_ACTIVE_HIGH>;
reset-gpios = <&lsio_gpio1 30 GPIO_ACTIVE_LOW>;
mclk = <24000000>;
mclk_source = <0>;
mipi_csi;

status = "okay";
port {
imx290_mipi_1_ep: endpoint {
remote-endpoint = <&mipi_csi1_ep>;
data-lanes = <1 2 3 4>;
link-frequencies = /bits/ 64 <148500000 222750000 297000000 445500000>;
clocks-lanes = <0>;
};
};
};
};

&isi_0 {
status = "okay";

cap_device {
status = "okay";
};

m2m_device {
status = "disabled";
};
};

&isi_1 {
status = "disabled";
};

&isi_2 {
status = "disabled";
};

&isi_3 {
status = "disabled";
};

&isi_4 {
status = "okay";

cap_device {
status = "okay";
};

m2m_device {
status = "disabled";
};
};

&isi_5 {
status = "disabled";
};

&isi_6 {
status = "disabled";
};

&isi_7 {
status = "disabled";
};

&mipi_csi_0 {
/delete-node/port@0;

/* Camera 0 MIPI CSI-2 (CSIS0) */
port@0 {
reg = <0>;
mipi_csi0_ep: endpoint {
remote-endpoint = <&imx290_mipi_0_ep>;
data-lanes = <1 2 3 4>;
};
};
};

&mipi_csi_1 {
/delete-node/port@1;

/* Camera 1 MIPI CSI-2 (CSIS1) */
port@1 {
reg = <1>;
mipi_csi1_ep: endpoint {
remote-endpoint = <&imx290_mipi_1_ep>;
data-lanes = <1 2 3 4>;
};
};
};

I have identified that the configuration including the various elements involved and the associated c files are as follows:

IMX290 -> mxc-mipi-csi2 -> mxc_isi -> mxc_isi.capture

imx290.c -> imx8-parallel-csi.c -> imx8-isi-hw.c -> imx8-isi-cap.c

imx290 5-001a -> mxc-mipi-csi2.0 -> mxc_isi.0 -> mxc_isi.0.capture
/dev/v4l-subdev5 -> /dev/v4l-subdev2 -> /dev/v4l-subdev0 -> /dev/video0

imx290 6-001a -> mxc-mipi-csi2.1 -> mxc_isi.4 -> mxc_isi.4.capture
/dev/v4l-subdev4 -> /dev/v4l-subdev3 -> /dev/v4l-subdev1 -> /dev/video1

Everything appears to be connecting and these are the relevant dmesg entries:

[ 1.104816] mxc-isi 58100000.isi: mxc_isi.0 registered successfully
[ 1.111512] mxc-isi 58140000.isi: mxc_isi.4 registered successfully
[ 1.135744] mxc-mipi-csi2 58227000.csi: lanes: 4, name: mxc-mipi-csi2.0
[ 1.159759] mxc-mipi-csi2 58247000.csi: lanes: 4, name: mxc-mipi-csi2.1
[ 1.654458] link-frequencies 0 value 148500000
[ 1.658916] link-frequencies 1 value 222750000
[ 1.663369] link-frequencies 2 value 297000000
[ 1.667814] link-frequencies 3 value 445500000
[ 1.672298] imx290 5-001a: 5-001a supply vdda not found, using dummy regulator
[ 1.679570] imx290 5-001a: 5-001a supply vddd not found, using dummy regulator
[ 1.686832] imx290 5-001a: 5-001a supply vdddo not found, using dummy regulator
[ 1.730977] link-frequencies 0 value 148500000
[ 1.735440] link-frequencies 1 value 222750000
[ 1.739886] link-frequencies 2 value 297000000
[ 1.744333] link-frequencies 3 value 445500000
[ 1.748805] imx290 6-001a: 6-001a supply vdda not found, using dummy regulator
[ 1.756075] imx290 6-001a: 6-001a supply vddd not found, using dummy regulator
[ 1.763333] imx290 6-001a: 6-001a supply vdddo not found, using dummy regulator
[ 1.946591] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[ 1.953157] mx8-img-md: Registered mxc_isi.4.capture as /dev/video1
[ 1.959593] mx8-img-md: Registered sensor subdevice: imx290 6-001a (1)
[ 1.966135] mx8-img-md: Registered sensor subdevice: imx290 5-001a (2)
[ 1.972674] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]
[ 1.979301] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[ 1.985754] mx8-img-md: created link [mxc_isi.4] => [mxc_isi.4.capture]
[ 1.992372] mx8-img-md: created link [mxc-mipi-csi2.1] => [mxc_isi.4]
[ 1.998821] mx8-img-md: created link [imx290 5-001a] => [mxc-mipi-csi2.0]
[ 2.005615] mx8-img-md: created link [imx290 6-001a] => [mxc-mipi-csi2.1]

Where I think I am being tripped up is the transfer of the Y10_1X10 image format specifically from the camera to the isi capture device (I want to capture the image in the Y10_1X10 format).

The I2C driver (imx290.c) is independent of actually transferring image data and I believe it is working correctly.

The ISI hardware (imx8-isi-hw.c) is set as per the previous example patch to bypass any image manipulation in the data transfer process:

/* set outbuf format */
val |= dst_fmt->color << CHNL_IMG_CTRL_FORMAT_OFFSET;

mxc_isi->cscen = 1;

if(dst_fmt->fourcc == V4L2_PIX_FMT_Y10)
{
goto bypass_csc;
}

if(dst_fmt->fourcc == V4L2_PIX_FMT_Y12)
{
goto bypass_csc;
}

if (is_yuv(src_fmt->fourcc) && is_rgb(dst_fmt->fourcc)) {
/* YUV2RGB */
csc = YUV2RGB;
/* YCbCr enable??? */
val |= (CHNL_IMG_CTRL_CSC_MODE_YCBCR2RGB << CHNL_IMG_CTRL_CSC_MODE_OFFSET);
val |= (CHNL_IMG_CTRL_YCBCR_MODE_ENABLE << CHNL_IMG_CTRL_YCBCR_MODE_OFFSET);
} else if (is_rgb(src_fmt->fourcc) && is_yuv(dst_fmt->fourcc)) {
/* RGB2YUV */
csc = RGB2YUV;
val |= (CHNL_IMG_CTRL_CSC_MODE_RGB2YCBCR << CHNL_IMG_CTRL_CSC_MODE_OFFSET);
} else {
/* Bypass CSC */
bypass_csc:
pr_info("bypass csc\n");
mxc_isi->cscen = 0;
val |= CHNL_IMG_CTRL_CSC_BYPASS_ENABLE;
}

That leaves the two controlling c files imx8-parallel-csi.c and imx8-isi-cap.c which collect the images from the camera and insert them into the ISI and then subsequently capture the images for the user.

I believe that the settings that I am using in these two files are incorrect because the only transfer formats that are available to me are in colour i.e. not a 10 (or 12) bit greyscale format.

In imx8-parallel-csi.c in the function:

static void mxc_pcsi_csr_config(struct mxc_parallel_csi_dev *pcsidev)

I have tried:

/* Config PL Data Type */
if (mf->code == MEDIA_BUS_FMT_Y10_1X10)
val = IF_CTRL_REG_DATA_TYPE(DATA_TYPE_OUT_RAW) | (1<<8);
else if (mf->code == MEDIA_BUS_FMT_Y12_1X12)
val = IF_CTRL_REG_DATA_TYPE(DATA_TYPE_OUT_RAW) | (1<<8);
else
val = IF_CTRL_REG_DATA_TYPE(DATA_TYPE_OUT_YUV444);
writel(val, pcsidev->csr_regs + IF_CTRL_REG_SET);

and:

/* Config CTRL REG */
val = readl(pcsidev->csr_regs + CSI_CTRL_REG);
if (mf->code==MEDIA_BUS_FMT_Y10_1X10)
val |= (CSI_CTRL_REG_DATA_TYPE_IN(DATA_TYPE_IN_BAYER_10BITS));
else if (mf->code==MEDIA_BUS_FMT_Y12_1X12)
val |= (CSI_CTRL_REG_DATA_TYPE_IN(DATA_TYPE_IN_BAYER_12BITS));
else
val |= (
CSI_CTRL_REG_DATA_TYPE_IN(DATA_TYPE_IN_UYVY_BT656_8BITS) |
CSI_CTRL_REG_HSYNC_POL |
CSI_CTRL_REG_MASK_VSYNC_COUNTER(3) |
CSI_CTRL_REG_HSYNC_PULSE(2));

But don't have any option to set the data types as monochrome:

#define DATA_TYPE_OUT_NULL (0x00)
#define DATA_TYPE_OUT_RGB (0x04)
#define DATA_TYPE_OUT_YUV444 (0x08)
#define DATA_TYPE_OUT_YYU420_ODD (0x10)
#define DATA_TYPE_OUT_YYU420_EVEN (0x12)
#define DATA_TYPE_OUT_YYY_ODD (0x18)
#define DATA_TYPE_OUT_UYVY_EVEN (0x1A)
#define DATA_TYPE_OUT_RAW (0x1C)

#define DATA_TYPE_IN_UYVY_BT656_8BITS 0x0
#define DATA_TYPE_IN_UYVY_BT656_10BITS 0x1
#define DATA_TYPE_IN_RGB_8BITS 0x2
#define DATA_TYPE_IN_BGR_8BITS 0x3
#define DATA_TYPE_IN_RGB_24BITS 0x4
#define DATA_TYPE_IN_YVYU_8BITS 0x5
#define DATA_TYPE_IN_YUV_8BITS 0x6
#define DATA_TYPE_IN_YVYU_16BITS 0x7
#define DATA_TYPE_IN_YUV_24BITS 0x8
#define DATA_TYPE_IN_BAYER_8BITS 0x9
#define DATA_TYPE_IN_BAYER_10BITS 0xA
#define DATA_TYPE_IN_BAYER_12BITS 0xB
#define DATA_TYPE_IN_BAYER_16BITS 0xC

Similarly in imx8-isi-cap.c I have added the following into:

struct mxc_isi_fmt mxc_isi_out_formats[]

{
.name = "BA10",
.fourcc = V4L2_PIX_FMT_Y10,
.depth = { 16 },
.color = MXC_ISI_OUT_FMT_RAW10,
.memplanes = 1,
.colplanes = 1,
.mbus_code = MEDIA_BUS_FMT_Y10_1X10,
}, {
.name = "BA12",
.fourcc = V4L2_PIX_FMT_Y12,
.depth = { 16 },
.color = MXC_ISI_OUT_FMT_RAW12,
.memplanes = 1,
.colplanes = 1,
.mbus_code = MEDIA_BUS_FMT_Y12_1X12,
}

But still only have:

/*
* Pixel link input format
*/
struct mxc_isi_fmt mxc_isi_src_formats[] = {
{
.name = "RGB32",
.fourcc = V4L2_PIX_FMT_RGB32,
.depth = { 32 },
.memplanes = 1,
.colplanes = 1,
}, {
.name = "YUV32 (X-Y-U-V)",
.fourcc = V4L2_PIX_FMT_YUV32,
.depth = { 32 },
.memplanes = 1,
.colplanes = 1,
}
};

Does anyone know what settings to use to correctly pass a Y10_1X10 formatted image through mx8-parallel-csi.c and imx8-isi-cap.c (or indeed other files if required) in order that I can capture it correctly?

Thank you very much in advance for your assistance!

[joanxie]

for raw10 data capture on imx8qm ISI, you need to shift 4bits, did you try this? for example, XXDD DDDD DDDD XXXX, D is for data

[hossein_veluon]

Hi,

As @Peter_ManyPebbles mentioned the input is set to "DATA_TYPE_IN_BAYER_10BITS " and the output is set to "DATA_TYPE_OUT_RAW". We save the raw data to file (see attachments) using the below terminal command:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat='Y10 ' --stream-mmap 2 --stream-count=1 --stream-to=test3.raw --verbose

When we check the raw data, the values of all pixels are like this DDDD DDXX XXXX XXXX.

If we do  'Y12 ' instead we get this DDDD DDDD XXXX XXXX.

This simply means that, we always are loosing 4 bits, but we are not sure which 4 bits!

Also according to this the Y10 should have "unused high bits padded with 0", which should mean a correct data should look like this: XXXX XXDD DDDD DDDD

[Peter_ManyPebbles · Accepted solution]

Ok, to tie this off, we did get it working nicely but with some bit shifting required on the output data as described by joanxie.

The solution allowed us to pass through both Y10_1X10 and Y12_1X12 image formats from the IMX290.

The patch referenced by the previous case that I referenced was applied but with the following modifications (please reference the patch file directly):

For the file: drivers/staging/media/imx/imx8-parallel-csi.c:

/* Config PL Data Type */
if (mf->code == MEDIA_BUS_FMT_Y8_1X8)
val = IF_CTRL_REG_DATA_TYPE(DATA_TYPE_OUT_RAW) | (1<<8);
else if (mf->code == MEDIA_BUS_FMT_Y10_1X10)
val = IF_CTRL_REG_DATA_TYPE(DATA_TYPE_OUT_RAW) | (1<<8);
else if (mf->code == MEDIA_BUS_FMT_Y12_1X12)
val = IF_CTRL_REG_DATA_TYPE(DATA_TYPE_OUT_RAW) | (1<<8);
else
val = IF_CTRL_REG_DATA_TYPE(DATA_TYPE_OUT_YUV444);
writel(val, pcsidev->csr_regs + IF_CTRL_REG_SET);

...

/* Config CTRL REG */
val = readl(pcsidev->csr_regs + CSI_CTRL_REG);
if (mf->code==MEDIA_BUS_FMT_Y8_1X8)
val |= (CSI_CTRL_REG_DATA_TYPE_IN(DATA_TYPE_IN_BAYER_16BITS));
else if (mf->code==MEDIA_BUS_FMT_Y10_1X10)
val |= (CSI_CTRL_REG_DATA_TYPE_IN(DATA_TYPE_IN_BAYER_16BITS));
else if (mf->code==MEDIA_BUS_FMT_Y12_1X12)
val |= (CSI_CTRL_REG_DATA_TYPE_IN(DATA_TYPE_IN_BAYER_16BITS));
else
val |= (
CSI_CTRL_REG_DATA_TYPE_IN(DATA_TYPE_IN_UYVY_BT656_8BITS) |
CSI_CTRL_REG_HSYNC_POL |
CSI_CTRL_REG_MASK_VSYNC_COUNTER(3) |
CSI_CTRL_REG_HSYNC_PULSE(2));

For the file: drivers/staging/media/imx/imx8-isi-hw.c The bypass was implemented as per the patch:

/* set outbuf format */
val |= dst_fmt->color << CHNL_IMG_CTRL_FORMAT_OFFSET;

mxc_isi->cscen = 1;

if(dst_fmt->fourcc == V4L2_PIX_FMT_GREY)
{
goto bypass_csc;
}

if(dst_fmt->fourcc == V4L2_PIX_FMT_Y10)
{
goto bypass_csc;
}

if(dst_fmt->fourcc == V4L2_PIX_FMT_Y12)
{
goto bypass_csc;
}

For the file: drivers/staging/media/imx/imx8-isi-cap.c the following output formats were added:

{
.name = "BA08",
.fourcc = V4L2_PIX_FMT_GREY,
.depth = { 8 },
.color = MXC_ISI_OUT_FMT_RAW8,
.memplanes = 1,
.colplanes = 1,
.mbus_code = MEDIA_BUS_FMT_Y8_1X8,
}, {
.name = "BA10",
.fourcc = V4L2_PIX_FMT_Y10,
.depth = { 16 },
.color = MXC_ISI_OUT_FMT_RAW16,
.memplanes = 1,
.colplanes = 1,
.mbus_code = MEDIA_BUS_FMT_Y10_1X10,
}, {
.name = "BA12",
.fourcc = V4L2_PIX_FMT_Y12,
.depth = { 16 },
.color = MXC_ISI_OUT_FMT_RAW16,
.memplanes = 1,
.colplanes = 1,
.mbus_code = MEDIA_BUS_FMT_Y12_1X12,
}

The input formats were not changed in this file (and BA08 output format was not tested).

Resulting from the above we received Y10_1X10 data in a 16 bit buffer in the format:

XXDD DDDD DDDD XXXX (where D is data)

We also received Y12_1X12 data in a 16 bit buffer in the format:

XXDD DDDD DDDD DDXX (where D is data)

I hope this benefits people!

[joanxie]

what format you set in the CHNL_IMG_CTRL? did you set 001100b - RAW10 - 10-bit RAW data packed into 16-bit WORD with 6 LSBs waste bits? pls double check it