[iMX8M-Mini] High speed MIPI CSI-2 camera sensor

[petkan]

Hello,

I wrote a driver Sony IMX492, which is high speed, Bayer 10/12bpp, ~47mpix sensor and i am currently having issues getting proper image out of it.  The sensor is streaming at 1.728gbps per lane and there's 4 of them.  This is above the 1.5gbps the D-PHY support so we had to lower the transmission to 1.44gbps, but still no joy.

Managed to get csi1_core and csi1_root_clk from 500mhz to 1ghz, but this caused the capture to hang.  I am not sure if this high frequency is supported by the CSI bridge, though.

My question is: has anyone here managed to get high speed, high resolution, 4 lane MIPI CSI2 sensor to work properly with iMX8M-Mini?  For the record: I am using Variscite Symphony carrier board with var-som-mx8m-mini.

cheers,

Petko

[smohsin]

There are many high speed MIPI CSI-2 camera sensors on the market. Some of the most popular ones include the Sony IMX258, Samsung S5K2P7, and OmniVision OV13850. These sensors can all capture high quality images at very high speeds, making them ideal for applications like video recording and surveillance.

[petkan]

Yes, both ends have all four lanes enabled.  What i can't find sufficient information about is:

1. What's the maximum clock rate between the MIPI-CSI and CSI bridge?
2. Is it possible to set the parallel bus between the above in 16bit (dual mode) for raw12 pixel formats?
3. I've found some apocryphal information that 32bit (quad mode) is broken for at least i.MX8M-Mini.  Comments?
4. What is the highest rate CSI-2 camera sensor that is known to work with i.MX8M-Mini on all 4 lanes?

thanks,

Petko

[jack_mao]

@petkan ,

   Here are some feedback

1. What's the maximum clock rate between the MIPI-CSI and CSI bridge?

[jack] maximum input clock to CSI bridge is 333MHz

1. Is it possible to set the parallel bus between the above in 16bit (dual mode) for raw12 pixel formats?

[jack] yes

1. I've found some apocryphal information that 32bit (quad mode) is broken for at least i.MX8M-Mini.  Comments?

[jack] Sorry, I don't see any recorded similar issue as your description, do you want to use 32bit  for rawx? But commnly we don't use 32bit for raw 12/16, it wastes bits.

1. What is the highest rate CSI-2 camera sensor that is known to work with i.MX8M-Mini on all 4 lanes?

[jack] we only have a reference EVK  sensor, we don't have the highest rate recording. According to the 333M pixel clock, any sensors  width x hight x framerate<333MHz will be support in theory.

[petkan]

Jack, thanks for the reply.

In case raw12 can be transferred in one sample, what would be the configuration on both (mipi and bridge) ends?  I've tried by setting PIXEL_MODE to 'dual' in mipi's MIPI_CSIS_ISP_CONFIG_CH0, and SENSOR_16BITS (cr3) and MIPI_DOUBLE_CMPNT (cr18) in bridge's setup.  This did not produce working setup.  Is there anything else that needs to be configured?

I was hoping it would be possible to pack two raw12 samples in 32bit (as two 16bit wors), but judging by your reply this may not be possible, correct?

thanks,

Petko

[jack_mao]

@petkan ,

       The 8m mini CSI support dual mode,  that means two samples for one pixel clock, we have verified dual pixel mode + yuv422 8bit, but not sure if it supports dual+12bit raw,  I will confirm with our SOC. 

      By the way, could you tell me your sensor clock, frame rate and resolution?

[petkan]

Hi Jack,

I've mentioned these details in my very first post, but here they are:

• 47mpix, Sony, imx492 sensor;
• 4 lanes, 1.728gbps per lane;
• max resolution: 8240 × 5628 active pixels;
  
• between 10 and 13 FPS depending on the resolution;

I am aware that yuv422 work in 16bit mode.  This is in fact the default driver setting.  However, i could not find proof of raw12 ever been successfully tested in this mode, nor i could program MIPI and Bridge so that streaming won't break.

thanks,

Petko

[jack_mao]

@petkan ,

      Thank your  feedback, your sensor resolution is far more than our 8mmini design target, our target is 1080p60. By the way our mipi lane is 1.5G  bps, if your input is larger than this value, then the input data should have issue.

      The others , could you tell me the sensor input clock to mipi, the input bpp should be 12(raw12), could you calculate the input bandwidth inputclock x 12, it should be less than 333M x 16.

      I 'm checking with the IP vendor and SOC to see if the chip can support dual mode with raw12.

[jack_mao]

Hi @petkan ,

    I just confirmed with SOC that 8m mini mipi-csi doesn't support dual mode for raw12, so you can't pack 2 12bit sample to 1 word.

[petkan]

                port {
                        /* MIPI CSI-2 bus endpoint */
                        imx492_to_mipi_csi2: endpoint {
                                remote-endpoint = <&imx8mm_mipi_csi_in>;
                                clock-lanes = <0>;
                                data-lanes = <1 2 3 4>;
                                link-frequencies = /bits/ 64 <745000000>;
                        };
                };

The link-frequency is invalid, because the PLL most likely can't produce this number and, what's important, the driver ignores this parameter.  I am able to set the aforementioned clocks in a different DT fragment. This is the imx492 part, this is the mipi-csi one:

                port@0 {                                                                                                                                                              
                        reg = <0>;                                                                                                                                                    
                                                                                                                                                                                      
                        imx8mm_mipi_csi_in: endpoint {                                                                                                                                
                                remote-endpoint = <&imx492_to_mipi_csi2>;                                                                                                             
                                data-lanes = <1 2 3 4>;
                        };
                };

 Have in mind these two pieces are from the mainline v5.19 kernel, this is why they might look unfamiliar to you. I'm asking about the practical speed limit between the mipi-csi and csi-bridge as that's where the link breaks.  Or at least that's what it looks like.  In case you can squeeze some more information, this is the relevant clock tree:

                                 enable  prepare  protect                                duty  hardware
   clock                          count    count    count        rate   accuracy phase  cycle    enable
-------------------------------------------------------------------------------------------------------
 sys_pll2                             1        1        0  1000000000          0     0  50000         Y
    sys_pll2_out                      5        5        0  1000000000          0     0  50000         Y
       sys_pll2_1000m                 3        3        0  1000000000          0     0  50000         Y
          csi1_phy_ref                1        1        0  1000000000          0     0  50000         Y
          csi1_core                   1        1        0   500000000          0     0  50000         Y
             csi1_root_clk            1        1        0   500000000          0     0  50000         Y

thanks,

Petko

[joanxie]

for high speed camera support, like 4k@30 capture, don't forget set in the dts file as below：

&csi1_bridge {
 fsl,two-8bit-sensor-mode;
 status = "okay";
};

 in the csi driver mx6s_capture.c, pls add this

static int mx6s_configure_csi(struct mx6s_csi_dev *csi_dev)
case V4L2_PIX_FMT_UYVY:
case V4L2_PIX_FMT_YUYV:
cr18 |= BIT_MIPI_DATA_FORMAT_YUV422_8B;
+ cr18 |= BIT_MIPI_DOUBLE_CMPNT;
break;
case V4L2_PIX_FMT_SBGGR8:
cr18 |= BIT_MIPI_DATA_FORMAT_RAW8;

[petkan]

Hi Joanxie,

I did set the MIPI_DOUBLE_CMPNT bit in CR18, but it does not work for RAW10/12 formats.  It effectively kills the communication between MIPI and bridge.  Could you please confirm (or deny) the above?

Could you also check what's the fastest RAW12 camera sensor that uses all four CSI lanes, that is known to work with iMX8M-Mini?

thanks,

Petko

[joanxie]

for raw10/raw12, did you add them in the csi driver mx6s_capture.c?? this driver only raw8 as default, you need add raw10 or raw12 format there

[joanxie]

yes, i.mx8mm doesn't support raw data in the default bsp, you can change it in the bsp, but imx8mm couldn't convert it after capture, you need add ISP module to convert raw data to yuv data, current csi driver mx6s_capture.c doesn't include raw10 format, did you add it in the driver?

[HAKANAYDN]

Hi joanxie, How can we add RAW10 format ?

[petkan]

Thanks for the pointer.  However, i've tried this already and the csi bridge has stopped receiving the stream.  As noted before, this sensor only supports raw10 and raw12 bayer format.  According to the manual, only YUV422 support two-byte setup.

Do you think this will also work with raw formats?

thanks,

Petko

[joanxie]

how did you set the dts file? pls share the settings, did you set 4 lane in the dts file already?