IMX8 MIPI CSI2 Weird artifacts every 4th pixel

[dpenebac]

My name is Dorien and am the firmware engineer primarily working on the MIPI CSI debugging for the IMX8.

While I probably can't share everything regarding source files and such, I do have a few questions and information that might point to something. 

Currently, we are able to receive test patterns from a custom test pattern FPGA. There are a few issues with the output raw images but we are currently using the ISI along with the ov5640 mipi driver and v4l2 to obtain "raw image" captures. Even though we have a few options, we are able to successfully receive "perfect" white images and black images (where white and black are simply raw10 packets with all 10 bits set high or low). 

However, when running what we are calling a "hgradient" pattern, OR any other set of bits that are not strictly white or strictly black, we start seeing some artifacts. We are currently receiving these test patterns as raw10 packets, our pipeline currently outputs the data shifted 4 bits to the right in LE format. For example the first pixel is white, the hex storage would be 0xF03F, after LE -> 0x3FF0 and shift by 4, 0x3FF, we get a 10 bit white pixel.  

After running v4l2-ctl --stream-mmap --stream-count=1 --stream-to=/tmp/frame.raw and transferring the image off over SCP. We use a little python script to rearrange the 10 bit test pattern to be opened using cv2 libraries. 

Output shows

bypass csc

Input fmt Y10P
Output fmt Y10

<

At a first glance, we can see the initial artifact. The image itself is a true gradient ONLY when removing every 4th pixel from the image. This also occurs with other test patterns as well.

The second and third artifact are a bit more nebulous.

For some reason, every second line's first 4 pixels are always some dark color. Not always zero, but just always really dark.

The last two lines are always black for some reason. However we have reason to believe this is an fpga test pattern generation bug but still something to call out.

Alongside all of that, I recently discovered that when sending NON white/black test patterns, we are receiving CRC errors by directly reading the RX CSI register space during streaming. 

I’m seeing an issue where every 4th byte of pixel data appears corrupted, but only on non-uniform images. White and black test patterns look fine. My first thought is that CRC errors could be the cause, but I don’t understand why CRC would ONLY fail on non-uniform.

From the hardware side we seem to be receiving MIPI control packets correctly since images can still be generated. I’m looking for ideas on what other possible error sources might cause the above artifact behavior, and the best ways to approach debugging it.

Regarding source code, we are using this as our format. 

    {

        .name = "RAW10P",

        .fourcc = V4L2_PIX_FMT_Y10,

        .depth = { 16 },

        .color = MXC_ISI_OUT_FMT_RAW10P,

        .memplanes = 1,

        .colplanes = 1,

        .mbus_code = MEDIA_BUS_FMT_Y10_1X10,

    }

where the output of the test pattern generator as mipi packets look like 

[joanxie]

different imx8 processors has different MIPI csi IP, so I need to know which imx8 processor you use? imx8qm? if you use raw10 capture, did you change the mipi csi driver to support it?

[dpenebac]

We are utilizing the IMx8x Quad Plus. 

I didn't make any modifications to the mipi csi driver exactly, but I have been modifying imx8isi-fmt.c and imx8-isi-fmt.c to modify the fourcc value and the output format value for the ISI drivers and confirmed via the output of v4l2. 

What modifications would I have to make to the csi driver? Currently, the driver that is seemingly loaded is the imx8-mipi-csi2.c attached below. 

[joanxie]

do you know which silicon revision of imx8qxp  you use? ISI has shift issue for raw data capture, I need to know what silicon revision you use

[dpenebac]

Hey, thanks for your help.

I will get back to you later today regarding the specific silicon revision, however, I believe we are already accounting for the bit shifting in our image generation post ISI. 

We currently utilize a python script to 

    # Load the entire RAW16 file as uint16 LE

    arr = np.fromfile(filename, dtype='<u2')

    # Convert 16-bit RAW10 stored in 16-bit words to 10-bit

    img10 = (arr >> 4).astype(np.uint16)  # 10-bit range: 0..1023

    # Scale 10-bit to 8-bit

    img8 = cv2.convertScaleAbs(img10, alpha=255.0 / 1023.0)

when we view the raw output via xxd or a hex editor, the white image shows

f0 3f f0 3f f0 3f

After post-processing of our script we do the following

f0 3f -> LE : 3ff0

3ff0 >> 4 : 3ff 

where 3FF is valid white data since it is 10 bits high which makes sense.

There are also some notes within the documentation which allude to this bit shifting

[joanxie]

yes, this is what I'm talking about, since you have this issue, after you capture the raw data, need shift it to get proper data, for raw10, you need shift 4bits to get correct data

[dpenebac]

Thanks for the clarification, the 4-bit shift requirement is something we already handle in post-processing. That still leaves a set of very repeatable problems that I haven't been able to get useful help on:

• Every 4th pixel is corrupted for non-uniform images (white/black patterns look fine).
• On alternating lines, the first 4 pixels are consistently much darker than they should be.
• The last two lines of every frame are always black (we suspect an FPGA test-pattern issue here).
• RX CSI CRC errors appear in the CSI registers only when the frame contains non-uniform data.

Could someone from the CSI/ISI/driver teams please explain exactly where RAW10 unpacking/bit shifting is expected to occur in the pipeline (CSI RX → ISI → DMA → v4l2 → userspace)? Specifically:

1. Which component/driver is expected to convert the MIPI RAW10 packet layout into the 16-bit words we see in memory?
2. Where should the 4-bit right shift (to extract 10-bit pixels) happen — in the CSI IP, in the ISI, in the imx8-mipi-csi2 driver, or later in v4l2/userland?
3. Are there known silicon revisions / ISI errata on i.MX8QXP (or related imx8qxp silicon revisions) that affect RAW10 alignment or cause CRC/noise only with non-uniform patterns?

To make the corruption clearer: RAW10 packs 4 pixels into 5 bytes (the low 8 bits of pixels 0–3 in bytes 0–3, and the top 2 bits of each pixel combined into the 5th byte). White or black frames hide any misalignment because all bits are identical; however for non-uniform patterns we see behavior consistent with the packed MSBs being misaligned. In our captures the MSB bits that should belong to the 4th pixel in the group appear to be distributed incorrectly across the group (effectively ending up in the LSB positions of neighboring pixels), which visually produces a corrupted every-4th-pixel pattern. In short: the group-of-4 unpacking seems wrong not just a single bit shift and so the corruption only shows up when the bits vary across pixels. We have been able to "unscramble" the image by reversing this formatting.In our captures, the MSBs from Byte4 appear misaligned. Instead of cleanly mapping the top 2 bits to each pixel, the “4th pixel” in each group looks corrupted, and its bits spill into neighboring pixels. This led us to realize that what was actually happening was the following unpacking pattern:If we were expecting

P1 = {P1[9:2], LSB[7:6]}
P2 = {P2[9:2], LSB[5:4]}
P3 = {P3[9:2], LSB[3:2]}
P4 = {P4[9:2], LSB[1:0]}

Instead of the expected standard unpacking we are actually seeing below (confirmed 100% via multiple different gradients):

P1 = {P1[9:2], P4[7:6]}
P2 = {P2[9:2], P4[5:4]}
P3 = {P3[9:2], P4[3:2]}
P4 = {LSB[7:0], P4[1:0]}

So effectively the top 2 bits from the 4th pixel (Byte4) were not correctly aligned to each pixel, which produces the observed “every 4th pixel corruption.” After identifying this, we were able to “unscramble” the image in software to recover the intended pattern.Do you know why we need to do this?One additional observation: this issue may be related to an endianness mismatch between the i.MX8 hardware and our MIPI transmitter (FPGA). In other words, the way the CSI IP reads the RAW10 5-byte groups might differ from how the FPGA packs them.For example, if we denote a RAW10 group as ABCDE → P1, P2, P3, P4, LSB bits, a potential mismatch could cause the i.MX8 to read it as something like A_CB_ED, effectively swapping or misaligning the LSB bits. This would explain why the 4th pixel in each group appears corrupted in non-uniform images.This is still speculative the exact behavior depends on both the CSI IP hardware and how strictly the MIPI spec is implemented by the transmitter but it could be worth thinking about.Thanks,
Dorien Penebacker

[joanxie]

firstly, you need confirm that your chip is B0 version or C0 version, for B0 which has shift issue, 

Oni.MX8QXP B0, when receiving RAW data from CSI, the data will always be shifted to the left for some RAW formats. Here are some examples: RAW10 {valid_data, 4’b0} . After right shift 4 bits, you will get the correct image.

On i.MX8QXP C0, there is a new bit that can fix this alignment, making all bits from received data align starting from bit 0. The table below shows the register and the new bit [31].

you also can refer to the link for raw data capture on imx8qxp

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX8QXP-capture-raw-bayer-data-and-deba...