How to set up the MIPI CSI-2 and ISI to transfer RAW10 images on the i.MX8QM

[gabrielaugustoc]

My work is based on Linux Kernel 4.14.170

I have succeeded to stream images through MIPI CSI-2 from a 16Mpx image sensor based on the following article: https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX8QXP-capture-raw-bayer-data-and-deba...

Also, I have created the device driver for this image sensor and changed other device drivers into /drivers/media/platform/imx8/ folder.

The ISI has been set to bypass the image format conversion (CSC), when the image sensor is configured as V4L2_PIX_FMT_SGRBG10 (mxc_isi_channel_set_csc function, in the mxc-isi-hw.c file)

The iMX8 is receiving bytes, the buffer size is correct, like for 640x480x10bits = 384000bytes, and it is possible to display the frames from the image sensor but some bytes are missing, like show in the example below: 00 14 00 34 00 14 00 34 00 3C 00 58 00 68 00 68 00 4C 00 44 00 38 00 38 00 60 00 54 00 34 00 54 00 34 00 24 00 6C 00 24 00 70 00 34 00 68 00 28 00 4C 00 48 00 4C 00 18 00 44 00 3C 00 44 00 54

It looks like the MIPI CSI-2 or Pixel Link is not transmitting the RAW10, just 8 bits.

How can I set up the MIPI CSI-2 and/or the Pixel Link for RAW10 bits?

[gabrielaugustoc · Accepted solution]

Hi everyone,

I got the solution for this issue:

{
.name = "BA10",
.fourcc = V4L2_PIX_FMT_SGRBG10,
.depth = { 16 },
.color = MXC_ISI_OUT_FMT_RAW16,
.memplanes = 1,
.colplanes = 1,
.mbus_code = MEDIA_BUS_FMT_SGRBG10_1X10,
}

After change ".color" parameter to "MXC_ISI_OUT_FMT_RAW16" the output buffer is completely filled with the 10bits RAW from the image sensor:

"B0 0B 80 0B B0 0B 80 0B 80 0B 40 0B 80 0B 90 0A 00 0B 90 0A 00 0B 70 0A 30 0A 60 0A 50 0A 30 0A"

Attached a sample image.

[mz-fixposition]

Hello @gabrielaugustoc 

Thanks for your post, it is very helpful!

Except for adding the RAW format to `mxc_isi_out_formats`, did you also had to add another format to `mxc_isi_src_formats`?

As you previously pointed out, the only formats that are natively in imx8-isi-cap.c are RGB4 and YUV4

/*
 * Pixel link input format
 */
struct mxc_isi_fmt mxc_isi_src_formats[] = {
	{
		.name		= "RGB32",
		.fourcc		= V4L2_PIX_FMT_RGB32,
		.depth		= { 32 },
		.memplanes	= 1,
		.colplanes	= 1,
	}, {
		.name		= "YUV32 (X-Y-U-V)",
		.fourcc		= V4L2_PIX_FMT_YUV32,
		.depth		= { 32 },
		.memplanes	= 1,
		.colplanes	= 1,
	}
};

 
If yes, what did you add here?

I am also interested in perersonhou's question, how did you read the raw buffer?

Best, Moritz

[mallibeerala]

Hi,

I am using RGB888 format, and what "mxc_isi_src_formats" I have to use, when I am changing these parameters no change is happening. Is their any other area we have to modify?

Thanks & Regards,
Mallikarjuna B.

[arnoutdiels]

I ran into the same question: How to get RAW10/RAW12 working on an IMX8, with the ISI.

Officially the datasheet states that when using the CSI peripheral to the ISI, only YUV or RGB can be used. But, like gabriel saw, by patching the code, this does seem to work.

The fundamental steps seemed to be:

- Make sure the MIPI format is correctly configured. (mipi_csis_subdev_init defaults this to mipi_csis_formats[0]. In my case, I changed it to MEDIA_BUS_FMT_SBGGR12_1X12 to get MIPI_CSIS_ISPCFG_FMT_RAW12)

- Make sure the ISI is in bypass, but even IF it is in bypass, the output format stills needs to be correct according to the datasheet. I added an entry in mxc_isi_src_formats, as wel as in mxc_isi_out_formats, and made sure isi_cap_fmt_init also took my output entry as default. (This indeed needed to be 16bit (MXC_ISI_OUT_FMT_RAW16) in order to be correct))

Even though this seems to pass the image through in RAW12 (stuffed in 16bit) correctly, my question to NXP is why is this officially not supported through the ISI?

Kind regards,

Arnout

[petersonhou]

hi,gabrielaugustoc

how could you find the frame data from image sensor lost some byte like you said": 00 14 00 34 00 14 00 34 00 3C 00 58 00 68 00 68 00 4C 00 44 00 38 00 38 00 60 00 54 00 34 00 54 00 34 00 24 00 6C 00 24 00 70 00 34 00 68 00 28 00 4C 00 48 00 4C 00 18 00 44 00 3C 00 44 00 54"? from the CSI2RX or ISI or  oscilloscope?

have you try to shift bit of the frame data as the RM mentioned?

[gabrielaugustoc]

Thanks for your response.
Yesterday I was reading the datasheet IMX8DQXPRM.pdf, page 4.925, and I changed the .depth = { 10 } to .depth = { 16 } on the imx8-isi-cap.c, as you have pointed, this change have increased the output buffer to 16bits (10bits RAW and 6bits padding with 0), that is nice .
But unfortunately, the issue still there, I have one byte 0x00 and the other byte with data.
I have attached my changes in the Linux kernel as a patch file.
I am using a Toradex board and I am working on the Toradex branch toradex_4.14-2.3.x-imx (git://git.toradex.com/linux-toradex.git).
Also, I have included here some debug message from the device drivers when I running an application to read images using V4L2:

[   81.125584] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[   87.843953] drivers/media/platform/imx8/mxc-isi-cap.c, mxc_isi_source_fmt_init, 989 CONFIG_VIDEO_ECAM not configured
[   87.862943] drivers/media/platform/imx8/mxc-isi-hw.c, mxc_isi_channel_set_csc, 300 dst_fmt->color = 0x000C
[   87.872730] drivers/media/platform/imx8/mxc-isi-hw.c, mxc_isi_channel_set_csc, 306 is using V4L2_PIX_FMT_SGRBG10
[   87.882993] drivers/media/platform/imx8/mxc-isi-hw.c, mxc_isi_channel_set_csc, 325, bypass csc
[   87.891645] input fmt RGB4
[   87.894394] output fmt BA10
[   87.897192] drivers/media/platform/imx8/mxc-isi-hw.c, mxc_isi_channel_config, 614, CHNL_CTRL_CHNL_BYPASS_ENABLE=1
[   88.258740] drivers/media/platform/imx8/mxc-mipi-csi2.c, mxc_mipi_csi2_hc_config, 240, MIPI Lane = 4
[   88.268003] drivers/media/platform/imx8/mxc-mipi-csi2.c, mxc_mipi_csi2_hc_config, 254, Virtual Channel = 0
[   88.277737] drivers/media/platform/imx8/mxc-mipi-csi2.c, mxc_mipi_csi2_csr_config, 207, CSI2SS_DATA_TYPE = 0xFFF7FFFF
[   88.288388] drivers/media/platform/imx8/mxc-mipi-csi2.c, mxc_mipi_csi2_csr_config, 219, hs_settle = 6
[   89.371574] i.mx8-hdp 56268000.hdmi: No EDID function, use default video mode
[   89.380368] i.mx8-hdp 56268000.hdmi: No EDID function, use default video mode

I am trying to figure out where the "input fmt" was set as RGB4, and if this is the correct for RAW10.
I am thinking the "output fmt" is correct as BA10 "RAW10", but the Pixel Link connection between the MIPI CSI-2 [1] and the ISI is not correct.
I will continue to read the datasheet and device drivers.

My best regards,

Gabriel

[weidong_sun]

Hi gabrielaugustoc ,

Try the setting, please!

In imx8-isi-cap.c, add lines below:

struct mxc_isi_fmt mxc_isi_out_formats[] = {
{
+ .name = "RAW10",
+ .fourcc = V4L2_PIX_FMT_SBGGR10,
+ .depth = { 16 },
+ .color = MXC_ISI_OUT_FMT_RAW10,
+ .memplanes = 1,
+ .colplanes = 1,
+ .mbus_code = MEDIA_BUS_FMT_SBGGR10_1X10,
+ },

B.R,

Weidong

[koilarulraj]

Hi @weidong_sun ,

We are trying to capture RAW10 (V4L2_PIX_FMT_SGRBG10) from the camera using iMX8Mplus platform.

We have tried to add the mentioned "mxc_isi_out_formats" in the platform driver. (Both RAW10 and BA16)

But when capturing the image, we receive the images with the required size but the data bytes are mostly 0x0. Please find the attached raw.

We have probed the MIPI signals from the camera and confirmed that we are receiving the proper MIPI activity.

Do we need to configure anything in the iMX8MPlus platform to get RAW10 format from the camera?