Issue with IMX93 MIPI CSI2: RAW10 custom camera hanged-up – no data captured

[caohieu]

Hello, I am currently deploying a system using the SB-UCM-iMX93-Compulab development board.
My project uses a custom camera (Onsemi-AR2020, 20MP, 5120×3840) with output format SBGGR RAW10.
The custom camera has its own clock, and does not require a reset or power-up trigger.

-  I am using Yocto-project to build a kernel module, which I forked from here:
https://github.com/compulab-yokneam/meta-bsp-imx9/blob/EVAL-UCM-iMX93-3.0/README.md

-  Previously, I used my camera on the RPi CM4. I then brought up the AR2020 on the i.MX93 after applying some fixes mentioned here:
https://www.virtanatech.com/blog/indoor-nano-drones-integrating-a-vio-capable-mipi-csi2-camera-on-th... 

-  And then, I modified Yocto-project with some patches, referencing here:
https://github.com/pleiades-br/meta-pleiades-canopus/tree/b58e3fcd92d0b502d0212208bc67da189b57243f/r... 

With my setup, I started the MIPI camera clock at a relatively low frequency to check whether it must be running. The value was-op_sys_clk_freq_mhz(MIPI_DATA_RATE): 149.4643 MHz.

However, it hung during image capture:

root:~# v4l2-ctl --device=/dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'RGBP' (16-bit RGB 5-6-5)
        [1]: 'RGB3' (24-bit RGB 8-8-8)
        [2]: 'BGR3' (24-bit BGR 8-8-8)
        [3]: 'YUYV' (YUYV 4:2:2)
        [4]: 'YUV4' (32-bit A/XYUV 8-8-8-8)
        [5]: 'NV12' (Y/UV 4:2:0)
        [6]: 'NM12' (Y/UV 4:2:0 (N-C))
        [7]: 'YM24' (Planar YUV 4:4:4 (N-C))
        [8]: 'XR24' (32-bit BGRX 8-8-8-8)
        [9]: 'AR24' (32-bit BGRA 8-8-8-8)
        [10]: 'BG10' (10-bit Bayer BGBG/GRGR)
                Size: Discrete 5120x3840


root:~# v4l2-ctl --device=/dev/video0 --set-fmt-video=width=5120,height=3840,pixelformat=BG10 && v4l2-ctl --device=/dev/video0 --stream-mmap --stream-count=1 --stream-to=output.raw --verbose
VIDIOC_QUERYCAP: ok
                VIDIOC_REQBUFS returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_G_FMT returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_STREAMON returned 0 (Success)
^C
--> So I had to cancel at this point.

Absolutely no data was captured.:

root:~# ls -l
total 0
-rw-r--r-- 1 root root 0 Feb 27 17:34 output.raw

And here is the log when I enable it:

破坏者

[ 3879.356978] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_g_selection
[ 3879.357152] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_g_fmt_mplane
[ 3879.357165] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357173] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357179] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357185] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357191] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357197] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357203] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357209] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357215] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357220] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357226] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_enum_fmt
[ 3879.357235] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_s_fmt_mplane, fmt=0x30314742
[ 3879.357241] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_fmt_try
[ 3879.357265] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_g_selection
[ 3879.357271] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_g_selection
[ 3879.371654] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_g_selection
[ 3879.371839] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_g_selection
[ 3879.371850] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_g_selection
[ 3879.372009] isi-capture 4ae40000.isi:cap_device: cap_vb2_queue_setup, buf_n=4, size=31457280
[ 3879.415864] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_g_fmt_mplane
[ 3879.415900] isi-capture 4ae40000.isi:cap_device: cap_vb2_buffer_prepare
[ 3879.415914] isi-capture 4ae40000.isi:cap_device: cap_vb2_buffer_prepare
[ 3879.415921] isi-capture 4ae40000.isi:cap_device: cap_vb2_buffer_prepare
[ 3879.415929] isi-capture 4ae40000.isi:cap_device: cap_vb2_buffer_prepare
[ 3879.415936] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_g_fmt_mplane
[ 3879.415945] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_streamon
[ 3879.415957] ar2020 2-0036: ar2020 Calling PowerOn - Kernel Module [1.0.9b]
[ 3879.424109] ar2020 2-0036: ar2020 Done PowerOnStep Without ERROR
[ 3879.433402] mxc-mipi-csi2.0: format: 0x3007
[ 3879.438423] /drivers/staging/media/imx/imx8-isi-hw.c, mxc_isi_channel_set_csc, 361 is using V4L2_PIX_FMT_SBGGR10
[ 3879.438433] bypass csc
[ 3879.438435] input fmt BG10
[ 3879.438438] output fmt BG10
[ 3879.438446] mxc-isi_v1 4ae40000.isi: input_size(5120,3840), output_size(4096,3840)
[ 3879.438465] isi-capture 4ae40000.isi:cap_device: cap_vb2_start_streaming
[ 3879.448967] isi-capture 4ae40000.isi:cap_device: cap_vb2_start_streaming: num_plane=0 discard_size=31457280 discard_buffer=00000000a3278a9c
[ 3879.449013] ar2020 2-0036: Starting stream
[ 3879.548107] ar2020 2-0036: CSI_LANE_MODE confirmed as 2-lane (0x01)
[ 3879.565623] ar2020 2-0036: FRAME_COUNT after streaming: 0xff
[ 3879.565641] ar2020 2-0036: Stream started
[ 3879.565649] dwc-mipi-csi2-host 4ae00000.csi: enter enable=1
[ 3879.565878] dwc-mipi-csi2-host 4ae00000.csi: data_type:0x2b, virtual chan: 0
[ 3879.565884] dwc-mipi-csi2-host 4ae00000.csi: hsa_time=0, hbp_time=0, hsd_time=0, hline_time=1280
[ 3879.565891] dwc-mipi-csi2-host 4ae00000.csi: ipi mode: camera, color_mode: color mode 16
[ 3879.565898] dwc-mipi-csi2-host 4ae00000.csi: format: 0x3007, w/h=(5120, 3840)
[ 3879.565906] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 Version: 0x3135302a
[ 3879.565910] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 lanes: 0x1
[ 3879.565915] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 HOST RESETN: 0x1
[ 3879.565919] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 INT STATUS MAIN: 0x0
[ 3879.565923] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DATA IDS1: 0x0
[ 3879.565927] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DATA IDS2: 0x0
[ 3879.565931] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DPHY CFG: 0x0
[ 3879.565935] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DPHY MODE: 0x0
[ 3879.565939] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 INT STATUS AP MAIN: 0x0
[ 3879.565943] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DATA IDS VC1: 0x0
[ 3879.565947] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DATA IDS VC2: 0x0
[ 3879.565951] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DPHY SHUTDOWN: 0x1
[ 3879.565955] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DPHY RESET: 0x1
[ 3879.565959] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DPHY RX STATUS: 0x10000
[ 3879.565963] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DPHY STOP STATUS: 0x0
[ 3879.565967] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DPHY TEST CTRL0: 0x0
[ 3879.565971] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 DPHY TEST CTRL1: 0x0
[ 3879.565975] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 PPI PG PATTERN HIGH: 0x0
[ 3879.565979] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 PPI PG PATTERN WIDTH: 0x0
[ 3879.565983] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 PPI PG CONFIG: 0x0
[ 3879.565987] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 PPI PG ENABLE: 0x0
[ 3879.565991] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 PPI PG STATUS: 0x0
[ 3879.565995] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 IPI MODE: 0x1010100
[ 3879.565999] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 IPI VCID: 0x0
[ 3879.566003] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 IPI DATA TYPE: 0x2b
[ 3879.566007] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 IPI SOFT RESET: 0x1
[ 3879.566012] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 IPI ADV FEATURE: 0x0
[ 3879.566016] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 INT_MSK_PHY_FATAL: 0x3
[ 3879.566020] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 INT_MSK_PKT_FATAL: 0x3
[ 3879.566024] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 INT_MSK_PHY: 0x3
[ 3879.566028] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 INT_MSK_IPI_FATAL: 0x7f
[ 3879.566032] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 INT_MSK_BNDRY_FRAME_FATAL: 0x0
[ 3879.566036] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 INT_MSK_SEQ_FRAME_FATAL: 0x0
[ 3879.566040] dwc-mipi-csi2-host 4ae00000.csi: DWC CSI2 INT_MSK_DATA_ID: 0x0
[ 3879.566045] dwc-mipi-csi2-host 4ae00000.csi: gasket: CAMERA MUX: 0x10158
[ 3879.566049] dwc-mipi-csi2-host 4ae00000.csi: gasket: CSI REG: 0x2161c
[ 3879.566054] dwc-mipi-csi2-host 4ae00000.csi: gasket: MIPI -> ISI pixel ctrl: 0xe07
[ 3879.566058] dwc-mipi-csi2-host 4ae00000.csi: gasket: MIPI -> ISI pixel cnt: 0x0
[ 3879.566063] dwc-mipi-csi2-host 4ae00000.csi: gasket: MIPI -> ISI line cnt: 0x0
[ 3879.566069] isi-capture 4ae40000.isi:cap_device: mxc_isi.0.capture is no v4l2 subdev

--> (I cancelled it here)
[ 3880.420390] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_streamoff
[ 3880.420412] isi-capture 4ae40000.isi:cap_device: cap_vb2_stop_streaming
[ 3880.423421] ar2020 2-0036: Stream stopped
[ 3880.423444] dwc-mipi-csi2-host 4ae00000.csi: enter enable=0
[ 3880.423605] isi-capture 4ae40000.isi:cap_device: mxc_isi.0.capture is no v4l2 subdev

root:~# media-ctl -p
Media controller API version 6.6.52

Media device information
------------------------
driver          mxc-md
model           FSL Capture Media Device
serial
bus info        platform:42800000.bus:camera
hw revision     0x0
driver version  6.6.52

Device topology
- entity 1: mxc_isi.0 (16 pads, 2 links, 0 routes)
            type V4L2 subdev subtype Unknown flags 0
        pad0: Sink
                <- "mxc-mipi-csi2.0":4 [ENABLED]
        pad1: Sink
        pad2: Sink
        pad3: Sink
        pad4: Sink
        pad5: Sink
        pad6: Sink
        pad7: Sink
        pad8: Sink
        pad9: Sink
        pad10: Sink
        pad11: Sink
        pad12: Source
                -> "mxc_isi.0.capture":0 [ENABLED]
        pad13: Source
        pad14: Source
        pad15: Sink

- entity 18: mxc_isi.0.capture (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video0
        pad0: Sink
                <- "mxc_isi.0":12 [ENABLED]

- entity 22: mxc-mipi-csi2.0 (8 pads, 2 links)
             type Node subtype V4L flags 0
             device node name /dev/v4l-subdev0
        pad0: Sink
                <- "ar2020 2-0036":0 [ENABLED,IMMUTABLE]
        pad1: Sink
        pad2: Sink
        pad3: Sink
        pad4: Source
                -> "mxc_isi.0":0 [ENABLED]
        pad5: Source
        pad6: Source
        pad7: Source

- entity 31: ar2020 2-0036 (1 pad, 1 link, 0 routes)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev1
        pad0: Source
                [stream:0 fmt:SBGGR10_1X10/5120x3840 field:none colorspace:raw xfer:none]
                -> "mxc-mipi-csi2.0":0 [ENABLED,IMMUTABLE]

Below is my device-tree:

破坏者

&cameradev {
        status = "okay";
};


/* I2C3 + AR2020 camera */
&lpi2c3 {
        #address-cells = <1>;
        #size-cells = <0>;
        clock-frequency = <400000>;
        pinctrl-names = "default", "sleep";
        pinctrl-0 = <&pinctrl_lpi2c3>;
        pinctrl-1 = <&pinctrl_lpi2c3>;
        status = "okay";
        ar2020: mipi@36 {
                compatible = "onsemi,ar2020";
                reg = <0x36>;
                clocks = <&clk IMX93_CLK_MIPI_PHY_CFG>;
                clock-names = "xvclk";
                clock-frequency = <27000000>;
                /* clock source sensor */
                mipi_csi;
                status = "okay";

                port {
                        ar2020_mipi_ep: endpoint {
                                remote-endpoint = <&mipi_csi_ep>;
                                clock-lanes = <0>;
                                data-lanes = <1 2>;
                                link-frequencies = /bits/ 64 <1500000000>;
                                clock-noncontinuous;
                        };
                };
        };
};


&isi_0 {
        status = "okay";

        cap_device {
                status = "okay";
        };
};


/* MIPI CSI interface */
&mipi_csi {
        #address-cells = <1>;
        #size-cells = <0>;
        status = "okay";

        port@0 {
                reg = <0>;
                mipi_csi_ep: endpoint {
                        remote-endpoint = <&ar2020_mipi_ep>;
                        data-lanes = <2>;
                        cfg-clk-range = <28>;
                        hs-clk-range = <0x16>;
                        bus-type = <4>;
                        clock-noncontinuous;
                };
        };
};

Any solution for this, or what can I do now to fix this issue?
I attached some Yocto-patches related to the project.
I would greatly appreciate any support or suggestions from you.
Sincerely, thank you everyone.

[IXM]

How can I convert SBGGR RAW10 to RGB/YUV on the i.MX93? Using bayer2rgb is very slow — 30 fps drops to 8 fps

[vprajapati]

Did you find any solution?

I am facing same issue and same log I am getting. Not able to capture frames from sensor.

[caohieu]

Hello,
Downgrading the kernel image to version 6.1.55 fixed my issue.