Direct MIPI-CSI2 to memory access on i.MX8MP

[stefan_wenninge]

Hi,

we are trying to utilize the full resolution of 5440x3648 of our IMX183 sensor using the i.MX8MP.

There are three approaches we considered:

ISP

The ISP on the i.MX8MP is limited to 4096x3072 according to the reference manual.
In our setup we got it working with a maximum resolution of 4080x3072.

This limitation seems to be a technical hardware-associated limit.
=> This approach will not work.

ISI

The reference manual reports ambiguous limits for the ISI on the i.MX8MP.

On the one hand it is said to support up to 4k resolution on the other hand the Channel Image Configuration register CHNL_IMG_CFG (13.4.5.1.5) technically allows for widths and heights of up to 8192.

=> Confirmation from NXP would be appreciated as to if the ISI would support receiving resolutions of 5440x3648.

CSI to mem

In the i.MX6 there seemed to have been a direct access route between the CSI receiver and the system memory:

See figure 6-1 in IMXLXRM Rev. L4.9.88_2.0.0-ga, 05/2018.
The newer i.MX6 reference manual also shows this path:

See figure 37-1 in IMX6DQRM Rev. 6, 05/2020.

The IMX Kernel even contains a driver "mxc_v4l2_capture" that seems to grab images directly from the CSI and offer them in a V4L2 capture device.

On the i.MX8MP there seems to be no direct way from CSI to memory:

See IMX8MPRM Rev. 3, 08/2024.

Data from the CSI port must pass through the ISI to get to memory.
Here is where further information about the ISIs capabilities is required: 
The reference manual says the ISIs processing channels can only handle up to 4k resolution (I am guessing the limiting factor is the width of 4096).

But there also seems to be a path inside the ISI where most of the processing is skipped and the images could go directly to memory:

Maybe this path is not limited by the max width of 4096 but allows the full limit of 8192 from Register CHNL_IMG_CFG?
I am not sure if I am interpreting figure 13-8 correctly.

So at the core, my questions are:

• Can the ISI pipe image data of resolution 5440x3648 through to the memory?
• Is there another way to pipe image data from the CSI directly to memory?

[stefan_wenninge]

I found another detail: The AN12857 "i.MX 8M Series MIPI Capture System" (Rev. 0 — 15 March 2023) has a quite definitve note in section 4.2.3 ISI width limitation:

This confirms, that an unprocessed camera stream can be up to 8192x8192 in resolution.

The note makes it seem that the 2K/4K width limit does only apply to camera streams that undergo scaling and color space conversion. These are exactly the two processing steps that can be skipped using the CHNL_BYPASS field in the CHNL_CTRL register.

It seems fair to assume that setting the CHNL_BYPASS to 1 will make the camera stream an "unprocessed" camera stream and will allow for a resolution of 5440x3648 to be passed through the ISI.

@joanxie Can NXP confirm the validity of this note in AN13857 and confirm my assumption to be true?

[stefan_wenninge]

I did some further digging in the manual and found there is actually an option to skip the processing in the ISI:

See 13.4.5.1.2 Channel Control Register (CHNL_CTRL).

However it seem the field CHAIN_BUF in the same register kills any hopes of driving the ISI at image widths greater than 4096:

There is only a documented option to chain two of the 2K line buffers together. So the 4K width limit seems to be hardcoded.

There is however one detail that makes me wonder: For chaining two buffers together one bit would have sufficed.
But the CHAIN_BUF register has two bits. Enough for two theoretical further options. Maybe to chain 3 and 4 line buffers together?
In the case of 4 2K line buffers being chained, the maximum width of 8K would nicely align with the maximum width resolution that can be set in the Channel Image Configuration (CHNL_IMG_CFG) register. The WIDTH field in that register has 13bits -> 8192 maximum value.

Conspiciously the additional two options of the CHAIN_BUF field in the CHNL_CTRL register are labelled "Reserved for future use".

@joanxie Could anyone at NXP confirm or deny to me if the additional "Reserved for future use" options in the CHAIN_BUF field are in fact in preparation to the possibility of chaining of 3 or 4 line buffers together?

[joanxie]

i.mx8mp only two ISI, not like imx8qm, which means the pixel link can be done for ISI 2 and chained to ISI 1, you can refer to the ISI driver, if width >2k, need use chain buffer to support 

if (frm->o_width > ISI_2K) {

val = readl(mxc_isi->regs + CHNL_CTRL);

val &= ~CHNL_CTRL_CHAIN_BUF_MASK;

val |= (CHNL_CTRL_CHAIN_BUF_2_CHAIN << CHNL_CTRL_CHAIN_BUF_OFFSET);

writel(val, mxc_isi->regs + CHNL_CTRL);

if (mxc_isi->chain)

regmap_write(mxc_isi->chain, CHNL_CTRL, CHNL_CTRL_CLK_EN_MASK);

mxc_isi->chain_buf = 1;

more detailed chain buffer settings, pls refer to the ISI driver

https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/drivers/staging/media/imx/imx8-isi-hw.c

we have verified 4k with ISI0, for other resolution, you can tested by yourself, but let me remind, the ISI0 max clock is 500Mhz

[joanxie]

Can the ISI pipe image data of resolution 5440x3648 through to the memory?

>current bsp limited the ISI width less than 2K in driver

Is there another way to pipe image data from the CSI directly to memory?

>do you mean you need capture video from CSI to the memory before send them to the ISI?

[stefan_wenninge]

"current bsp limited the ISI width less than 2K in driver"
Section 13.4.2 of IMX8MPRM (Rev. 3, 08/2024) mentions the capability to chain two line buffers from two channels together to increase the line buffer limit from 2k to 4k.

In my experience this seems to be the case. I was able to receive images up to 4080x3648 via ISI.

"do you mean you need capture video from CSI to the memory before send them to the ISI?"
That is what I would like to do, yes.
My best guess would be figure 13-8 in my post which seems to show a way to pipe images through the ISI, but not through the processing pipeline within the ISI.

Do you have any more information on this "direct to memory" path within the ISI?