Confusion of MIPI-CSI2, ISI drivers for IMX8MP

[DWSmith]

Hello all.  I hope I can get some advice.

I am trying to get a Sony IMX287 (global shutter, 720x540, monochrome sensor) working with IMX8MP.  I wrote a new driver for the sensor based off the imx296 driver--modified to support this sensor and to provide the pads that the imx8_media_dev module expects.   Testing camera signals with a scope (LVDS CLK & Data, Vsync, Hsync, INCK...all are correct and valid).

We are using a Lattice FPGA to convert from the sensor's LVDS output to 4-lane MIPI-CSI2.  We had some issues with the LVDS intake and Sony parsing in the FPGA, but we've gotten past that and gotten some pixels through (a few lines).

Now, I enabled debug in the CSI2 driver (imx8-mipi-csi2-sam) and we see no errors at that level on either the FPGA or i.mx8 side (as seen from the CSI2 registers dump).

The problem is that our sensor outputs monochrome RAW10 & the CSI2 & ISI drivers don't seem to support that.

Consulting several threads here, I modified the drivers to support Y10_1X10 (RAW10).  v4l2-ctl can show the update list of formats, and when I start trying to capture a raw image, it correctly bypasses the csc, it gets the formats correct, and everything in the ISI registers appears correct.  But MIPI-CSI2 is giving FIFO overrun errors continuously, so it seems the ISI is not emptying the FIFO.  I'm not sure why.  But this led me down a path that got me confused.

The drivers I'm using come from the linux/drivers/staging/media/imx directory (to get the imx8_media_dev module).  

I was perplexed by the fact that I had to add support to the CSI2 and ISI drivers for RAW10 support, and that others have been having to do so for years.  I also have been confused by the kernel config options presented (and selected to build by default in NXPs Yocto setup).  There are other drivers under linux/drivers/media/platform/imx8.  There is a CSI2-YAV "yet another version" driver.  Then under linux/drivers/media/platform/nxp there is another csi2 driver, and another ISI driver.  These drivers appear to be much more recently maintained and already look to have support for RAW10.  But it isn't clear whether they work for the 8M Plus...  Or whether there is something extra that needs to be built to establish the links like the imx8_media_dev driver does.

Can anyone explain which drivers are the correct / best ones to use on the IMX8MP?

Does anyone have an idea why when I finally got all the drivers to agree on the MBUS format and PIXEL format, the ISI doesn't empty the FIFO for the CSI2?

I know from seeing other posts that the question will be "why aren't you using the ISP?"

There are a couple of reasons:

1) Setting up a driver for the ISP look incredibly complicated based on the porting guide.

2) We don't need any of the ISPs functions.  All we need is to capture RAW10 frames from the sensor and shove them into Gstreamer quickly.  All subsequent processing of the frames is done on a PC with Nvidia GPUs.  The ability to stream at a high frame-rate is the main thing we have to achieve.

3) While I have a functional i2c driver that could theoretically generate the .drv piece, there remains the camera calibration metadata that the ISP requires.  This is a non-trivial thing, and the rest of the product is not ready to go through such a process (nor do we have the time to commit to that for essentially no benefit).

3) Judging from the issues I've encountered trying to use the MIPI and ISI drivers so far, I have little confidence I won't run into similar issues going the ISP route.

Since we don't need the functions provided by the ISP, going through the ISI seemed to be the most straightforward option.

Any ideas?

[DWSmith]

To update on the current status.  We're not completely there yet, but getting much closer.

It turns out that the MIPI_CSIS_ISPCFG_ALIGN_32BIT and MIPI_CSIS_CMN_CTRL_HDR_MODE have nothing to do with the issue.

In fact, while setting MIPI_CSIS_ISPCFG_ALIGN_32BIT seemed to help by alleviating the FIFO overruns, it actually caused the malformatted data and led to 2 weeks of frustration.  The magical undocumented "HDR_MODE" bit doesn't seem to make a difference, from all I can tell.

I did some work on the clocks in the device tree that got past some CRC errors we were seeing.  I tried quad mode and dual mode, because the manual seemed to imply that one or the other was needed.  As they seem incompatible with 32-bit ("parallel") mode, I turned it off.  And I got correct pixel data (after using the ISI format for RAW16...RAW10 is left-shifted 6 bits just as the manual explains in that it places the data in the MSB of the bytes...using RAW16 gets it low-aligned as is preferred).  Getting the clocks a little better setup allowed me to get an image without FIFO overruns or CRC errors.  But, Quad-mode gave me a quarter-width image that only had the 1st of each group of 4 pixels.  Dual-mode gave me a half-width image with pixels 0 & 2.  They were the correct values, just missing data (which explains why the images were 1/4 and 1/2 width respectively).  The explanations for these modes in the manual is completely counter-intuitive.  It leads one to believe that something like RAW10 should require Quad-Mode.  No.  You want "normal"/"single pixel" mode.  And fast-enough clocks to pull the data out.

A conversation with ChatGPT revealed that it knows a lot more about what "Quad-pixel mode, Dual-pixel mode, and 32-bit mode" are than anything in NXP's documentation or even the MIPI-CSI2 spec I've looked at.

I still have to get my clocks cranked up to get to the frame-rate I need, but I'm getting somewhere now.

I am confused about the right clock settings in the device tree.

I *think* I need the following in the "endpoint" stanza:

link-frequencies = /bits/ 64 <2376000000>, <1188000000>;   // (total bandwidth for full & half rate modes)
max-lane-frequency = /bits/ 64 <297000000>;
max-pixel-frequency = /bits/ 64 <297000000>; // assuming MIPI-CSI2 "pixels" of 8-bits, even though our source pixels are 10-bit
max-data-rate = /bits/ 64 <2376000000>;

Currently using the default 500MHz for mipi-csi clocks and 200MHz for APB clock.

I believe I need to get the D-PHY clock up to 1188MHz (within the 1.5GHz max).

Pixel-clock / "wrap" clock...GPT thinks it needs to be 1.7GHz (720w * 540h * 437fps * 10bpp).  This is beyond the 1.5GHz max, so I may need to sacrifice some frame-rate.  Not sure if this number is correct...it looks like it is counting bits, not pixels.  So true pixel-rate might be just 170MHz.

Any advise on proper device-tree clock setup would be appreciated.  For reference:

Sensor is 720x540 active pixels, 297MHz per lane (DDR, so 594Mbps per lane), 4 lanes, total bandwidth in full-rate mode of 2.376Gbps.  Supports up to 437fps at full-rate.  I can adjust frame rate down by adjusting blanking.  We need a minimum of 250fps for our application.

Thanks for any advice on clocking, and I hope this is helpful for anyone else struggling with this.  It appears the ISI is in fact capable of bringing in RAW10 monochrome.

[joanxie]

suggest that you can refer to this AN

i.MX 8M Series MIPI Capture System (nxp.com)

[DWSmith · Accepted solution]

This didn't download anything, so I'm not sure what it contains.  Is there an AN number I can look up?

I got past the major issues.  We are now capturing and the next challenge is to keep from dropping frames in v4l2-ctl or gstreamer due to frame-rate.  The MIPI interface seems to do well up to 360fps (and can probably handle even higher frame-rate once we get the FPGA optimized).  

I see v4l2-ctl report a matching framerate to what the stats from the CSI events show, but it is dropping 170fps when I'm trying to capture to a file on ramdisk.  It seems to be v4l2-ctl that is dropping frames...seems like it cannot keep up.  Gstreamer seems to peak out around 100fps and I'm not sure why.  At this point, I'm not sure if it is too many interrupts, memory throughput, or just the userspace apps not able to keep up with the rate.

As for making RAW10 work with the ISI, for anyone else attempting this, the keys are:

1) add support for the input & output formats to the csi2-sam & isi drivers.  These drivers also have some bugs--some output formats are swapped and the gasket register dump code is double-applying the offset, so it reads the wrong address.  I will see if I can submit a patch.

2) set colorspace to COLORSPACE_RAW.  There are I think 3 places this has to be done.

3) DO NOT use 32-bit/Parallel mode.  RAW10 is aligned on 40-bits.  So you need "normal/single-pixel" mode.

4) ISI output format can be set to several different things.  If you set it to RAW10, it will be left-justified (6 LSBs padded), which is generally not what you want.  Set it to RAW16 and you will get the right output (16bit pixels with the upper bits padded with zeros).  You can get Y10 output.  If you add GREY format support (Y8), that works too & is then compatible with things like libjpeg.  Y16 is effectively the same as Y10.  

5) If you are going through an FPGA conversion like I am, make sure that the FPGA is getting the word-count right in the MIPI framer.  This was a further issue we had that produced CRC errors and FIFO overruns due to misalignment.

I'm going to mark this solved.

[DWSmith]

It is set to MEDIA_BUS_FMT_Y10_1X10 (0x200a).  Are you expecting that it should be MEDIA_BUS_FMT_SBGGR10_1X10?  It is set to Y10 because my addition of Y10 mode in mipi_csis_format.

Gasket is set to 2b (RAW10, same as MIPI-CSI2).

Debug output:

[15733.393411] mxc-mipi-csi2.0: mipi_csis_s_stream: 1, state: 0x0
[15733.393550] mxc-mipi-csi2.0: mipi_csis_imx8mp_phy_reset: bus fmt is 10 bit !
[15733.393582] disp_mix_gasket_config: set gasket format: 0x2b00, width: 728, height: 554
[15733.401521] mxc-mipi-csi2.0: fmt: 0x200a, 728 x 554
[15733.401531] disp_mix_gasket_enable: gasket enable = 1
[15733.421095] mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
[15733.425629] mxc-isi_v1 32e00000.isi: mxc_isi_irq_handler, LINE Received stat=0x40000100
[15733.426172] mxc-mipi-csi2.0: CSIS_VERSION[0]: 0x03060301
[15733.434171] mxc-isi_v1 32e00000.isi: mxc_isi_irq_handler, LINE Received stat=0x60000200
[15733.434178] mxc-isi_v1 32e00000.isi: mxc_isi_irq_handler, FRAME Received stat=0x60000200
[15733.442262] mxc-mipi-csi2.0: CSIS_CMN_CTRL[4]: 0x00004305
[15733.450306] mxc-mipi-csi2.0: CSIS_CLK_CTRL[8]: 0x000f0000
[15733.450314] mxc-mipi-csi2.0: CSIS_INTMSK[10]: 0x0fffff1f
[15733.450319] mxc-mipi-csi2.0: CSIS_INTSRC[14]: 0x00000000
[15733.450325] mxc-mipi-csi2.0: CSIS_DPHYSTATUS[20]: 0x000000f1
[15733.450330] mxc-mipi-csi2.0: CSIS_DPHYCTRL[24]: 0x0d80001f
[15733.450336] mxc-mipi-csi2.0: CSIS_DPHYBCTRL_L[30]: 0x000001f4
[15733.450342] mxc-mipi-csi2.0: CSIS_DPHYBCTRL_H[34]: 0x00000000
[15733.450360] mxc-mipi-csi2.0: CSIS_DPHYSCTRL_L[38]: 0x00000000
[15733.450366] mxc-mipi-csi2.0: CSIS_DPHYSCTRL_H[3c]: 0x00000000
[15733.450371] mxc-mipi-csi2.0: CSIS_ISPCONFIG_CH0[40]: 0x000008ac
[15733.450377] mxc-mipi-csi2.0: CSIS_ISPCONFIG_CH1[50]: 0x000008fd
[15733.450382] mxc-mipi-csi2.0: CSIS_ISPCONFIG_CH2[60]: 0x000008fe
[15733.450388] mxc-mipi-csi2.0: CSIS_ISPCONFIG_CH3[70]: 0x000008ff
[15733.450393] mxc-mipi-csi2.0: CSIS_ISPRESOL_CH0[44]: 0x022a02d8
[15733.450399] mxc-mipi-csi2.0: CSIS_ISPRESOL_CH1[54]: 0x80008000
[15733.450404] mxc-mipi-csi2.0: CSIS_ISPRESOL_CH2[64]: 0x80008000
[15733.450410] mxc-mipi-csi2.0: CSIS_ISPRESOL_CH3[74]: 0x80008000
[15733.450415] mxc-mipi-csi2.0: CSIS_ISPSYNC_CH0[48]: 0x00000000
[15733.450421] mxc-mipi-csi2.0: CSIS_ISPSYNC_CH1[58]: 0x00000000
[15733.450426] mxc-mipi-csi2.0: CSIS_ISPSYNC_CH2[68]: 0x00000000
[15733.450432] mxc-mipi-csi2.0: CSIS_ISPSYNC_CH3[78]: 0x00000000
[15733.450437] mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
[15733.450443] mxc-mipi-csi2.0: GPR_GASKET_0_CTRL[0]: 0x00002b01
[15733.450449] mxc-mipi-csi2.0: GPR_GASKET_0_HSIZE[4]: 0x000002d8
[15733.450455] mxc-mipi-csi2.0: GPR_GASKET_0_VSIZE[8]: 0x0000022a
[15733.450461] mxc-mipi-csi2.0: GASKET_0_ISI_PIX_CNT[1c]: 0x00000000
[15733.450467] mxc-mipi-csi2.0: GASKET_0_ISI_LINE_CNT[20]: 0x0000022a

[DWSmith]

Thank you.  Last night, I started getting some more realistic data after adding more debug output (hoping that doesn't mean there is a timing issue) to the ISI.  Also, the CSI driver's dump of gasket registers has a bug...it is reading the wrong offset & therefore always shows zeros.  I've seen someone else have that problem before & I discovered why.

I had seen a "reserved" setting and wondered what it was.  I definitely don't need HDR mode set.  Disabling that and setting ISPCFG_ALIGN_32BIT helped a lot.

For test, we have implemented a test-pattern in the FPGA that sends out pixel data like 0x004 0x008 0x00c 0x010 0x014 etc. to generate a gradient for line data (starts over each line).  We now get a gradient, but the greyscale color values are shifted and there are vertical stripes every 5th pixel.

The data above is coming out:

00 00 AA 01 01 02 02 02 03 03 04 00 AA 01 05 02 06 02 07 03 08 00 AA 01 09 02 0A 02 0B 03 0C 00 AA 01 0D 02 0E 02 0F 03 10 00 AA 01 11 02 12 02 13 03 14 00 AA 01 15 02 16 02 17 03 18 00 AA 01 19 02 1A 02 1B 03 1C 00 AA 01 1D 02 1E 02 1F 03 20 00 AA 01 21 02 22 02 23 03 24 00 AA 01 25 02 26 02 27 03 28 00 AA 01 29 02 2A 02 2B 03 2C 00 AA 01 2D 02 2E 02 2F 03 30 00 AA 01 31 02 32 02 33 03 34 00 AA 01 35 02 36 02 37 03 38 00 AA 01

Note that every 5th 16-bit word is 0xAA01.  If you right-shift the other pixels by 6, then you get the correct data.  So it seems like it is aligning the pixels high, rather than low.  Is there a way to fix it without having to filter the image after capture?  We expect to run at a high frame rate (>120fps) and I fear having to shift all pixels in software will impact our performance.  Also, every 5th word being 0xAA01 is interesting, since as I understand it, RAW10 on MIPI (format 0x2b) is formatted pixA2_9 pixB2_9 pixC2_9 pixD2_9 LSBs(a0a1b0b1c0c1d0d1) in 5-byte sequences.  So it seems like maybe the CSI receiver isn't reformatting the MIPI stream back to a RAW10/Y10 media bus format.

Any ideas?  Thank you.

[DWSmith]

An additional note: I did also update the CSI funcs disp_mix_gasket_config and mipi_csis_imx8mp_phy_reset with my added format in the same manner as the existing formats.

[joanxie]

this driver path is for imx8mp

drivers/staging/media/imx/imx8-mipi-csi2-sam.c

and besides of add raw10 in the mipi csi driver, did you add this in the ISI driver?  and tell me what bsp version you use

[DWSmith]

Thank you for responding.

We are on 6.6-nanbield.

Yes, I also added support for Y10 (RAW10) to the ISI.

I have enabled debug and the register settings look correct to me.

For CSI, I added to struct mipi_csis_formats:

{
.code = MEDIA_BUS_FMT_Y10_1X10,
.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW10,
.data_alignment = 16,
}

For imx8-isi-fmt.c, in struct mxc_isi_out_formats I added:

{
.name = "Y10 ",
.fourcc = V4L2_PIX_FMT_Y10,
.depth = { 16 },
.color = MXC_ISI_OUT_FMT_RAW16,  // also tried MXC_ISI_OUT_FMT_RAW10 to no avail
.memplanes = 1,
.colplanes = 1,
.align = 3,  // question...is this correct? manual shows RAW16 is packed into 32-bit DWORD
.mbus_code = MEDIA_BUS_FMT_Y10_1X10,
}

to imx8-isi-cap.c struct mxc_isi_src_formats I added:

{
.name = "Y10 ",
.fourcd = V4L2_PIX_FMT_Y10,
.depth = { 16 },
.memplanes = 1,
.colplanes = 1,
.align = 0, // is this correct? Incoming data is a stream of 10-bit pixels... // align=3 didn't work, align=1?
}

and I adjusted the function mxc_isi_get_src_fmt to recognize the added Y10 format and select its associated index (2).

I chose the ISI out format of RAW16, since I had seen another thread that said it needed to be RAW16 to work.  Regardless, I tried RAW10 with the same issue of FIFO overrun.

In debug output, I see that the csc is bypassed as it should be.  It selects my added format and when doing a raw capture with v4l-ctrl, it says: "bypass csc", "input fmt Y10", "output fmt Y10".

CHNL_IMG_CTRL register is 0x0F000001 which means CSC bypassed, RAW16 packed into 32-bit DWORD.  I tried setting above formats for RAW10 with .align = 1 (16-bit alignment) and got the same results, but CHNL_IMG_CTRL is 0x0C000001 (which is CSC bypassed, RAW10 packed into 16-bit DWORD with 6 waste bits).  All registers look correct, but the CSI2 reports all data as FIFO overrun.

I have checked my clock configurations and they all seem like they should work.

The sensor outputs 4-lanes of LVDS at 297MHz each lane, DDR.  This goes to an FPGA which converts to MIPI-CSI2 4-lanes at 297MHz each, DDR.  Bitrate is thus 297M*2(DDR)*4(lanes) = 2376Mbps (2.376Gbps).  Pixel clock should be 2376000000 / 10 (10bit pixels)= 237.6MHz.  Right?

By my understanding, the device-tree "clock-frequency" parameter is the "wrap" frequency and must exceed the pixel rate.  So the default of 500MHz should do it, but I have tried bumping that way up to 1188MHz and even the full 2376MHz with no change.

Oddly, if I use the driver as it comes and allow it to default to treating the data as RGB, I get pixel data captured, but it of course has color artifacts.

Here is some output:

root@crow:~# media-ctl -p -d0
Media controller API version 6.6.3

Media device information
------------------------
driver mxc-md
model FSL Capture Media Device
serial
bus info platform:32c00000.bus:camera
hw revision 0x0
driver version 6.6.3

Device topology
- entity 1: mxc_isi.0 (16 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
pad0: Sink
<- "mxc-mipi-csi2.0":4 [ENABLED]
pad1: Sink
pad2: Sink
pad3: Sink
pad4: Sink
pad5: Sink
pad6: Sink
pad7: Sink
pad8: Sink
pad9: Sink
pad10: Sink
pad11: Sink
pad12: Source
-> "mxc_isi.0.capture":0 [ENABLED]
pad13: Source
pad14: Source
pad15: Sink

- entity 18: mxc_isi.0.capture (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video0
pad0: Sink
<- "mxc_isi.0":12 [ENABLED]

- entity 22: mxc-mipi-csi2.0 (8 pads, 2 links)
type Node subtype V4L flags 0
device node name /dev/v4l-subdev0
pad0: Sink
<- "pregius 1-001a":0 [ENABLED,IMMUTABLE]
pad1: Sink
pad2: Sink
pad3: Sink
pad4: Source
-> "mxc_isi.0":0 [ENABLED]
pad5: Source
pad6: Source
pad7: Source

- entity 31: pregius 1-001a (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev1
pad0: Source
[fmt:Y10_1X10/720x540 field:none colorspace:raw xfer:none quantization:full-range
crop.bounds:(0,0)/720x540
crop:(0,0)/720x540]
-> "mxc-mipi-csi2.0":0 [ENABLED,IMMUTABLE]

root@crow:~# echo 2 > /sys/module/imx8_mipi_csi2_sam/parameters/debug
root@crow:~# v4l2-ctl --device /dev/video0 --set-fmt-video width=720,height=540,pixelformat=0x20303159 --stream-mmap --stream-to /tmp/testcap.raw --stream-count 1
[ 44.026944] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_g_selection
[ 44.033902] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_g_fmt_mplane
[ 44.040720] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.047190] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.053651] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.060126] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.066584] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.073055] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.079514] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.085987] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.092459] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.098917] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.105386] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.111859] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 44.118334] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_s_fmt_mplane, fmt=0x20303159
[ 44.126536] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_fmt_try
[ 44.132973] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_g_selection
[ 44.139706] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_g_selection
[ 44.146547] isi-capture 32e00000.isi:cap_device: cap_vb2_queue_setup, buf_n=4, size=777600
[ 44.156251] isi-capture 32e00000.isi:cap_device: cap_vb2_buffer_prepare
[ 44.162904] isi-capture 32e00000.isi:cap_device: cap_vb2_buffer_prepare
[ 44.169552] isi-capture 32e00000.isi:cap_device: cap_vb2_buffer_prepare
[ 44.176213] isi-capture 32e00000.isi:cap_device: cap_vb2_buffer_prepare
[ 44.182866] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_g_fmt_mplane
[ 44.189681] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_streamon
[ 44.209431] bypass csc
[ 44.211807] input fmt Y10
[ 44.214515] output fmt Y10
[ 44.217372] isi-capture 32e00000.isi:cap_device: cap_vb2_start_streaming
[ 44.224546] isi-capture 32e00000.isi:cap_device: cap_vb2_start_streaming: num_plane=0 discard_size=778240 discard_buffer=0000000087eba1c1
[ 44.236925] mxc-isi_v1 32e00000.isi: ISI CHNLC register dump, isi0
[ 44.243124] mxc-isi_v1 32e00000.isi: CHNL_CTRL[0x00]: e0000000
[ 44.249949] mxc-isi_v1 32e00000.isi: CHNL_IMG_CTRL[0x04]: f000001
[ 44.256676] mxc-isi_v1 32e00000.isi: CHNL_OUT_BUF_CTRL[0x08]: 7c707
[ 44.263226] mxc-isi_v1 32e00000.isi: CHNL_IMG_CFG[0x0c]: 21c02d0
[ 44.269950] mxc-isi_v1 32e00000.isi: CHNL_IER[0x10]: 3cfc0000
[ 44.276760] mxc-isi_v1 32e00000.isi: CHNL_STS[0x14]: 100
[ 44.283130] mxc-isi_v1 32e00000.isi: CHNL_SCALE_FACTOR[0x18]: 10001000
[ 44.289941] mxc-isi_v1 32e00000.isi: CHNL_SCALE_OFFSET[0x1c]: 00
[ 44.296254] mxc-isi_v1 32e00000.isi: CHNL_CROP_ULC[0x20]: 00
[ 44.302553] mxc-isi_v1 32e00000.isi: CHNL_CROP_LRC[0x24]: 00
[ 44.308839] mxc-isi_v1 32e00000.isi: CHNL_CSC_COEFF0[0x28]: 00
[ 44.315129] mxc-isi_v1 32e00000.isi: CHNL_CSC_COEFF1[0x2c]: 00
[ 44.321412] mxc-isi_v1 32e00000.isi: CHNL_CSC_COEFF2[0x30]: 00
[ 44.327698] mxc-isi_v1 32e00000.isi: CHNL_CSC_COEFF3[0x34]: 00
[ 44.333984] mxc-isi_v1 32e00000.isi: CHNL_CSC_COEFF4[0x38]: 00
[ 44.340270] mxc-isi_v1 32e00000.isi: CHNL_CSC_COEFF5[0x3c]: 00
[ 44.346559] mxc-isi_v1 32e00000.isi: CHNL_ROI_0_ALPHA[0x40]: 00
[ 44.352847] mxc-isi_v1 32e00000.isi: CHNL_ROI_0_ULC[0x44]: 00
[ 44.359133] mxc-isi_v1 32e00000.isi: CHNL_ROI_0_LRC[0x48]: 00
[ 44.365416] mxc-isi_v1 32e00000.isi: CHNL_ROI_1_ALPHA[0x4c]: 00
[ 44.371716] mxc-isi_v1 32e00000.isi: CHNL_ROI_1_ULC[0x50]: 00
[ 44.378001] mxc-isi_v1 32e00000.isi: CHNL_ROI_1_LRC[0x54]: 00
[ 44.384318] mxc-isi_v1 32e00000.isi: CHNL_ROI_2_ALPHA[0x58]: 00
[ 44.390618] mxc-isi_v1 32e00000.isi: CHNL_ROI_2_ULC[0x5c]: 00
[ 44.396910] mxc-isi_v1 32e00000.isi: CHNL_ROI_2_LRC[0x60]: 00
[ 44.403201] mxc-isi_v1 32e00000.isi: CHNL_ROI_3_ALPHA[0x64]: 00
[ 44.409486] mxc-isi_v1 32e00000.isi: CHNL_ROI_3_ULC[0x68]: 00
[ 44.415772] mxc-isi_v1 32e00000.isi: CHNL_ROI_3_LRC[0x6c]: 00
[ 44.422058] mxc-isi_v1 32e00000.isi: CHNL_OUT_BUF1_ADDR_Y[0x70]: c4900000
[ 44.428861] mxc-isi_v1 32e00000.isi: CHNL_OUT_BUF1_ADDR_U[0x74]: 00
[ 44.435148] mxc-isi_v1 32e00000.isi: CHNL_OUT_BUF1_ADDR_V[0x78]: 00
[ 44.441432] mxc-isi_v1 32e00000.isi: CHNL_OUT_BUF_PITCH[0x7c]: 5a0
[ 44.447807] mxc-isi_v1 32e00000.isi: CHNL_IN_BUF_ADDR[0x80]: 00
[ 44.454093] mxc-isi_v1 32e00000.isi: CHNL_IN_BUF_PITCH[0x84]: 00
[ 44.460376] mxc-isi_v1 32e00000.isi: CHNL_MEM_RD_CTRL[0x88]: 00
[ 44.466659] mxc-isi_v1 32e00000.isi: CHNL_OUT_BUF2_ADDR_Y[0x8c]: c4500000
[ 44.473464] mxc-isi_v1 32e00000.isi: CHNL_OUT_BUF2_ADDR_U[0x90]: 00
[ 44.479753] mxc-isi_v1 32e00000.isi: CHNL_OUT_BUF2_ADDR_V[0x94]: 00
[ 44.486054] mxc-isi_v1 32e00000.isi: CHNL_SCL_IMG_CFG[0x98]: 21c02d0
[ 44.492784] mxc-isi_v1 32e00000.isi: CHNL_FLOW_CTRL[0x9c]: 00
[ 44.530380] pregius 1-001a: setting VMAX to 586
[ 44.535304] pregius 1-001a: Setting HMAX to 1100
[ 44.564485] isi-capture 32e00000.isi:cap_device: mxc_isi.0.capture is no v4l2 subdev
[ 44.572275] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_g_fmt_mplane
[ 44.600285] systemd-journald[131]: /dev/kmsg buffer overrun, some messages lost.
[ 44.608024] systemd-journald[131]: /dev/kmsg buffer overrun, some messages lost.
[ 44.615547] systemd-journald[131]: /dev/kmsg buffer overrun, some messages lost.
^C[ 46.807024] isi-capture 32e00000.isi:cap_device: mxc_isi_cap_streamoff
[ 46.807033] isi-capture 32e00000.isi:cap_device: cap_vb2_stop_streaming
[ 46.820957] mxc-mipi-csi2.0: Frame End events: 1
[ 46.825576] mxc-mipi-csi2.0: Frame Start events: 1
[ 46.830366] mxc-mipi-csi2.0: Non-image data after odd frame events: 0
[ 46.836806] mxc-mipi-csi2.0: Non-image data before odd frame events: 0
[ 46.843331] mxc-mipi-csi2.0: Non-image data after even frame events: 0
[ 46.849858] mxc-mipi-csi2.0: Non-image data before even frame events: 0
[ 46.856470] mxc-mipi-csi2.0: Unknown Error events: 0
[ 46.861432] mxc-mipi-csi2.0: CRC Error events: 0
[ 46.866220] mxc-mipi-csi2.0: ECC Error events: 0
[ 46.870835] mxc-mipi-csi2.0: FIFO Overflow Error events: 141304
[ 46.876752] mxc-mipi-csi2.0: Lost Frame End Error events: 0
[ 46.882323] mxc-mipi-csi2.0: Lost Frame Start Error events: 87
[ 46.888156] mxc-mipi-csi2.0: SOT Error events: 0
[ 46.892851] isi-capture 32e00000.isi:cap_device: mxc_isi.0.capture is no v4l2 subdev

[joanxie]

in the mipi csi driver, did you enable MIPI_CSIS_ISPCFG_ALIGN_32BIT? if no, pls try to enable this bit and try to remove" val |= MIPI_CSIS_CMN_CTRL_HDR_MODE;"

[DWSmith]

I replied last night, I don't know why it is not showing up now.

The previous night, I started gaining some success in not having the FIFO overruns after adding some extra debug output (which is concerning about timing) and fixing a bug in the CSI driver's gasket register dump that was always showing all zeros for all gasket registers because it was reading the wrong offset (it applies the offset to a value that already includes the offset).  I fixed the gasket dump and saw that the gasket was getting data.

After receiving your response, I disabled the HDR bit.  I had noticed some "reserved" bits being set and wondered what that was.  I definitely don't need HDR mode.  

Setting ISPCFG_ALIGN_32BIT helped a lot.  We have implemented a simple gradient test-pattern in the FPGA to remove the actual CMOS sensor from the equation temporarily.  Now, we are seeing the gradient but with 2 issues:

1) The values are left-shifted 6 bits (so it would seem the 10-bit pixel data is going into the high end of the words instead of the low end).

2) We have vertical stripes because every 5th pixel is a constant value that doesn't correlate to our data.

#2 is striking to me as it fits the pattern of MIPI sending RAW10 by packing 4 10-bit pixels in 5 bytes.  According to the MIPI spec, it does this by sending pixA2-9 pixB2-9 pixC2-9 pixD2-9 followed by a 5th byte composed of the LSBs: pA0 pA1 pB0 pB1 pC0 pC1 pD0 pD1.  So it seems like this structure is still being preserved.

I thought the MIPI decoder would decode it back to 10-bit pixels, aligned in the lower bits of a 16-bit word ("Y10 format") and since we have the CSC bypassed, the ISI would write that straight to memory.

So for data that we send that looks like:

0x000 0x004 0x008 0x00c 0x010 0x014 etc. we get data that looks like:

00 00 AA 01 01 02 02 02 03 03 04 00 AA 01 05 02 06 02 07 03 08 00 AA 01 09 02 0A 02 0B 03 0C 00 AA 01 0D 02 0E 02 0F 03 10 00 AA 01 11 02 12 02 13 03 14 00 AA 01 15 02 16 02 17 03 18 00 AA 01 19 02 1A 02 1B 03 1C 00 AA 01 1D 02 1E 02 1F 03 20 00 AA 01 21 02 22 02 23 03 24 00 AA 01

The "AA01" words are the "5th" word that appears and causes our vertical stripes in the image.  You can see that if you right-shift by 6 the words "0101" "0202" etc. then you recover the data that was actually sent.

Is there a way to avoid having to have a post-capture filter?  Can it store the data correctly?  Do you have any ideas on how to correct this?

I am concerned about throughput if we need to add a filter.  We intend to run at low-resolution, but high frame-rate (120fps +).

Thank you!

[joanxie]

 in the function mxc_isi_cap_g_fmt_mplane of isi driver

https://github.com/nxp-imx/linux-imx/blob/lf-6.6.y/drivers/staging/media/imx/imx8-isi-cap.c#L846

did you change the V4L2_COLORSPACE_SRGB to V4L2_COLORSPACE_RAW?

[DWSmith]

Yes, I had already seen that and tried it.  The problem still occurs.

I am not sure how some people have gotten it to work.  It seems that the ISI wants to treat everything as RGB32 or YUV32 no matter how it came in, and that the CSI receiver does not reformat the received data.

If I configure things such that the CSI & ISI think that I'm sending RAW8, and I have it output RAW8 (but the camera and FPGA are outputting RAW10), then I see captured exactly what was on the MIPI bus and it makes sense:

Transmitted pixels:

Pix0: 0x000, Pix1: 0x004, Pix2: 0x008, Pix3: 0x00C, Pix4: 0x010, Pix5: 0x014, Pix6: 0x018, Pix7: 0x01C

Output:

0x00 0x01 0x02 0x03 0x00 0x05 0x06 0x07 0x08 0x00

The pixels are right-shifted 2 bits, and the 5th byte for each group of 4 pixels is the LSBs of the prior 4 10-bit values.

So I know that the FPGA is properly framing RAW10 on the CSI-2.

But when I set CSI & ISI to RAW10, and try to capture Y10 format for the same input, I get:

In ascending order: 0x0001 0x0000 0x0700 0x0A03 0x0D02 0x1001 0x0000 0x1700 0x1A03 0x1D02

byte reversing the 16-bit words (Y10 is supposed to by low-byte, high-byte order):

0x1000 0x0000 0x0007 0x030A 0x020D 0x0110 0x0000 0x0017 0x031A 0x021D

...we now get a 5th 16-bit word of 0x0000 (though the initial one seems to occur too soon).

It's even strange if we look at it as 32-bit words.

Indeed the manual seems to imply that if configured as RAW10, it will place the pixel value in the upper bits of the resulting 16-bit word (needing a right-shift of 6 bits to recover the correct data).  The data still look incorrect.

The online raw viewer at photopea.com seems to be able to make sense of the format, as it produces the gradient we are generating, with vertical black stripes as the 5th byte is zeros (LSBs).

We unfortunately cannot just use RAW8.  The FPGA IP only allows use of RAW10 or RAW12.

I am concerned that RAW functionality doesn't work in the ISI, though it seems some people have made it work.

I am looking at the ISP, but the instructions are not very clear and it seems I would somehow need to have a calibration done whether we need the lens dewarp or not. 

The changes to my i2c driver look fairly straightforward.  The architecture of the vvcam stuff seems overly complex to me...I'm not sure why it is required to have both a userspace and kernel space driver.  I am starting down that path, but it seems I have some weeks of wasted time if the ISI won't work.

The project is very concerned about the ability to utilize the camera solution on this processor.

We simply need to get our RAW10 camera output to come cleanly through the ISI into a raw file without affecting the pixel values or adding trash data from how MIPI transforms the data.  Then, we can do our processing off-board on a PC.  Can you summarize what you think is necessary to do this, and I can check against what I have done?

Thank you.

[joanxie]

in the function mipi_csis_set_fmt of mipi csi driver, do you mind print the "mf->code"? to check if mf->code is set to MEDIA_BUS_FMT_SBGGR10_1X10?