imx8mp with sony imx412 and 4 lanes

pmliquify · ‎02-29-2024

Hi all,

Situation:

I developed drivers for the sony imx412 and imx327 camera modules. Both cameras are working fine with 2 lanes. The imx327 additionally works fine with 4 lanes. All tests are done on both camera ports of the imx8mp.

See my device tree files attached!

imx8mp.dtsi
+-> imx8mp-var-dart.dtsi
    +-> imx8mp-var-dart-dt8mcustomboard.dts
        +-> imx8mp-var-dart-dt8mcustomboard-vc-mipi-isi0.dts

Problem:

The imx412 doesn't even start streaming with 4 lanes. On neither camera port. Sometimes I get a first image. After that the stream stops and never delivers a second image. This first image contains half of all data. The bayer pattern seems "broken". I guess this is because only every second line of the image is transmitted.

Update: My latest tests

Sony IMX412 from Vision Components on Variscite imx8mp Evaluation Kit, Kernel 5.15.71
2 + 4 lanes always with pixelformat RG10

With 2 lanes I do get valid images with full resolution 4032x3040 and also with smaller resolutions. Everything is fine with 2 lanes.

With 4 lanes I found that I get valid images up to an image resolution of 2780x3040. When I increase the image width to 2781 and above the image isn't valid anymore. See example images.

In all cases I do get following log

mxc-mipi-csi2.0: Frame End events: 2
mxc-mipi-csi2.0: Frame Start events: 2
mxc-mipi-csi2.0: Non-image data after odd frame events: 0
mxc-mipi-csi2.0: Non-image data before odd frame events: 0
mxc-mipi-csi2.0: Non-image data after even frame events: 0
mxc-mipi-csi2.0: Non-image data before even frame events: 0
mxc-mipi-csi2.0: Unknown Error events: 0
mxc-mipi-csi2.0: CRC Error events: 0
mxc-mipi-csi2.0: ECC Error events: 0
mxc-mipi-csi2.0: FIFO Overflow Error events: 0
mxc-mipi-csi2.0: Lost Frame End Error events: 0
mxc-mipi-csi2.0: Lost Frame Start Error events: 0
mxc-mipi-csi2.0: SOT Error events: 0

This leads me to believe, that the CSI receiver works fine, but the ISI is doing something bad.
Any suggestions?

pmliquify · ‎03-19-2024

Any help is highly appreciated. Any suggestions or advice on how to analyze the problem?

joanxie · ‎03-20-2024

did you try the settings I suggest to you one week ago?

pmliquify · ‎03-13-2024

Update: My latest tests

Sony IMX412 from Vision Components on Variscite imx8mp Evaluation Kit, Kernel 5.15.71
2 + 4 lanes always with pixelformat RG10

With 2 lanes I do get valid images with full resolution 4032x3040 and also with smaller resolutions. Everything is fine with 2 lanes.

With 4 lanes I found that I get valid images up to an image resolution of 2780x3040. When I increase the image width to 2781 and above the image isn't valid anymore. See example images.

In all cases I do get following log

mxc-mipi-csi2.0: Frame End events: 2
mxc-mipi-csi2.0: Frame Start events: 2
mxc-mipi-csi2.0: Non-image data after odd frame events: 0
mxc-mipi-csi2.0: Non-image data before odd frame events: 0
mxc-mipi-csi2.0: Non-image data after even frame events: 0
mxc-mipi-csi2.0: Non-image data before even frame events: 0
mxc-mipi-csi2.0: Unknown Error events: 0
mxc-mipi-csi2.0: CRC Error events: 0
mxc-mipi-csi2.0: ECC Error events: 0
mxc-mipi-csi2.0: FIFO Overflow Error events: 0
mxc-mipi-csi2.0: Lost Frame End Error events: 0
mxc-mipi-csi2.0: Lost Frame Start Error events: 0
mxc-mipi-csi2.0: SOT Error events: 0

This leads me to believe, that the CSI receiver works fine, but the ISI is doing something bad.
Any suggestions?

joanxie · ‎03-03-2024

could you share

imx8mp-var-dart-dt8mcustomboard.dts

and all of other include dtsi files? check if the clock is available for 4 data lane

pmliquify · ‎03-04-2024

Yes of coarse, I attached the files to my first post. See above.

joanxie · ‎03-06-2024

what format do you use for capture? not raw data? and what command do you use to capture the data from imx412?

pmliquify · ‎03-06-2024

I do capture images with RAW10 (RG10) pixelformat. The screenshot is created with my self develop test app (https://github.com/pmliquify/v4l2-test). The test app debayers the image with opencv.

But I do get the capture problem as well with v4l2-ctl.

joanxie · ‎03-11-2024

since you use raw data, why don't you use ISP for this? if you need to use ISI, do you add raw10 format in the ISI driver?

https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/drivers/staging/media/imx/imx8-isi-fmt.c

pmliquify · ‎03-11-2024

Hi joanxie,

I will use the ISP once the system delivers images via the v4l2 interface. My current problem is, that I do not receive images in memory with the IMX412 and 4 lanes. I DO get images with 2 lanes but NOT with 4 lanes.

Which possibilities exist to debug my system?
How is it possible to see/debug if the csi receivers receives data?
How is it possible to see if the isi is receiving data?
Is it possible that I do not receive data, because of wrong csis-hs-settle, csis-clk-settle settings? How to determine the correct settings?

Any help is highly appreciated!

joanxie · ‎03-12-2024

for 2 data lane and 4 data lane, do you use the same video format, resolution? just change the data lane from 2 to 4? could you give me the logfile when you use capture command with data lane 4? also pls dump the mipi csi registers to check, for csis-hs-settle, csis-clk-settle, which is depends on your actual mipi frequency, you can try to set csis-hs-settle = <16>;

pmliquify · ‎03-27-2024

I set csis-hs-settle = <16>; => No success.

Here is the reg dump for 2 lanes:

vc-mipi-cam 1-001a: vc_sen_set_binning_mode(): Set binning mode: 0
vc-mipi-cam 1-001a: vc_core_set_format(): Set format 0x300f (RG10)
vc-mipi-cam 1-001a: vc_core_try_format(): Try format 0x300f (RG10, format: 0x2b)
vc-mipi-cam 1-001a: vc_core_set_frame_size(): Set frame size (width: 4032, height: 3040)
vc-mipi-cam 1-001a: vc_core_get_format(): Get format 0x300f (RG10)
vc-mipi-cam 1-001a: vc_core_get_frame(): Get frame (width: 4032, height: 3040)
bypass csc
input fmt RGB4
output fmt RG10
i2c 1-0010: vc_mod_set_mode(): Set module mode: 0 (lanes: 2, format: RG10, type: STREAM)
i2c 1-0010: vc_mod_set_power(): Set module power: down
i2c 1-0010: vc_mod_set_power(): Set module power: up
vc-mipi-cam 1-001a: vc_sen_set_exposure(): Set sensor exposure: 10000 us
vc-mipi-cam 1-001a: vc_sen_set_gain(): Set sensor gain: 0
vc-mipi-cam 1-001a: vc_sen_set_blacklevel(): Set sensor black level: 64
vc-mipi-cam 1-001a: vc_sen_start_stream(): Start streaming
mxc-mipi-csi2.0: mipi_csis_s_stream: 1, state: 0x0
vc-mipi-cam 1-001a: vc_core_get_format(): Get format 0x300f (RG10)
mxc-mipi-csi2.0: mipi_csis_imx8mp_phy_reset: bus fmt is 10 bit !
mxc-mipi-csi2.0: fmt: 0x300f, 4032 x 3040
mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
mxc-mipi-csi2.0:         CSIS_VERSION[0]: 0x03060301
mxc-mipi-csi2.0:        CSIS_CMN_CTRL[4]: 0x00004905
mxc-mipi-csi2.0:        CSIS_CLK_CTRL[8]: 0x000f0000
mxc-mipi-csi2.0:          CSIS_INTMSK[10]: 0x0fffff1f
mxc-mipi-csi2.0:          CSIS_INTSRC[14]: 0x00000000
mxc-mipi-csi2.0:      CSIS_DPHYSTATUS[20]: 0x000000c0
mxc-mipi-csi2.0:        CSIS_DPHYCTRL[24]: 0x10800007
mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_L[30]: 0x000001f4
mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_H[34]: 0x00000000
mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_L[38]: 0x00000000
mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_H[3c]: 0x00000000
mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH0[40]: 0x000000ac
mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH1[50]: 0x000008fd
mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH2[60]: 0x000008fe
mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH3[70]: 0x000008ff
mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH0[44]: 0x0be00fc0
mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH1[54]: 0x80008000
mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH2[64]: 0x80008000
mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH3[74]: 0x80008000
mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH0[48]: 0x00000000
mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH1[58]: 0x00000000
mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH2[68]: 0x00000000
mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH3[78]: 0x00000000
mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
mxc-mipi-csi2.0:    GPR_GASKET_0_CTRL[60]: 0x00000000
mxc-mipi-csi2.0:   GPR_GASKET_0_HSIZE[64]: 0x00000000
mxc-mipi-csi2.0:   GPR_GASKET_0_VSIZE[68]: 0x00000000
vc-mipi-cam 1-001a: vc_sen_stop_stream(): Stop streaming
mxc-mipi-csi2.0: mipi_csis_s_stream: 0, state: 0x0
mxc-mipi-csi2.0: Frame End events: 5
mxc-mipi-csi2.0: Frame Start events: 5
mxc-mipi-csi2.0: Non-image data after odd frame events: 0
mxc-mipi-csi2.0: Non-image data before odd frame events: 0
mxc-mipi-csi2.0: Non-image data after even frame events: 0
mxc-mipi-csi2.0: Non-image data before even frame events: 0
mxc-mipi-csi2.0: Unknown Error events: 0
mxc-mipi-csi2.0: CRC Error events: 0
mxc-mipi-csi2.0: ECC Error events: 0
mxc-mipi-csi2.0: FIFO Overflow Error events: 0
mxc-mipi-csi2.0: Lost Frame End Error events: 0
mxc-mipi-csi2.0: Lost Frame Start Error events: 1190
mxc-mipi-csi2.0: SOT Error events: 1

And for 4 lanes:

vc-mipi-cam 1-001a: vc_sen_set_binning_mode(): Set binning mode: 0
vc-mipi-cam 1-001a: vc_core_set_format(): Set format 0x300f (RG10)
vc-mipi-cam 1-001a: vc_core_try_format(): Try format 0x300f (RG10, format: 0x2b)
vc-mipi-cam 1-001a: vc_core_set_frame_size(): Set frame size (width: 4032, height: 3040)
vc-mipi-cam 1-001a: vc_core_get_format(): Get format 0x300f (RG10)
vc-mipi-cam 1-001a: vc_core_get_frame(): Get frame (width: 4032, height: 3040)
bypass csc
input fmt RGB4
output fmt RG10
i2c 1-0010: vc_mod_set_mode(): Set module mode: 1 (lanes: 4, format: RG10, type: STREAM)
i2c 1-0010: vc_mod_set_power(): Set module power: down
i2c 1-0010: vc_mod_set_power(): Set module power: up
vc-mipi-cam 1-001a: vc_sen_set_exposure(): Set sensor exposure: 10000 us
vc-mipi-cam 1-001a: vc_sen_set_gain(): Set sensor gain: 0
vc-mipi-cam 1-001a: vc_sen_set_blacklevel(): Set sensor black level: 64
vc-mipi-cam 1-001a: vc_sen_start_stream(): Start streaming
mxc-mipi-csi2.0: mipi_csis_s_stream: 1, state: 0x0
vc-mipi-cam 1-001a: vc_core_get_format(): Get format 0x300f (RG10)
mxc-mipi-csi2.0: mipi_csis_imx8mp_phy_reset: bus fmt is 10 bit !
mxc-mipi-csi2.0: fmt: 0x300f, 4032 x 3040
mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
mxc-mipi-csi2.0:         CSIS_VERSION[0]: 0x03060301
mxc-mipi-csi2.0:        CSIS_CMN_CTRL[4]: 0x00004b05
mxc-mipi-csi2.0:        CSIS_CLK_CTRL[8]: 0x000f0000
mxc-mipi-csi2.0:          CSIS_INTMSK[10]: 0x0fffff1f
mxc-mipi-csi2.0:          CSIS_INTSRC[14]: 0x00000000
mxc-mipi-csi2.0:      CSIS_DPHYSTATUS[20]: 0x00000000
mxc-mipi-csi2.0:        CSIS_DPHYCTRL[24]: 0x1080001f
mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_L[30]: 0x000001f4
mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_H[34]: 0x00000000
mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_L[38]: 0x00000000
mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_H[3c]: 0x00000000
mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH0[40]: 0x000000ac
mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH1[50]: 0x000008fd
mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH2[60]: 0x000008fe
mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH3[70]: 0x000008ff
mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH0[44]: 0x0be00fc0
mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH1[54]: 0x80008000
mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH2[64]: 0x80008000
mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH3[74]: 0x80008000
mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH0[48]: 0x00000000
mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH1[58]: 0x00000000
mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH2[68]: 0x00000000
mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH3[78]: 0x00000000
mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
mxc-mipi-csi2.0:    GPR_GASKET_0_CTRL[60]: 0x00000000
mxc-mipi-csi2.0:   GPR_GASKET_0_HSIZE[64]: 0x00000000
mxc-mipi-csi2.0:   GPR_GASKET_0_VSIZE[68]: 0x00000000
vc-mipi-cam 1-001a: vc_sen_stop_stream(): Stop streaming
mxc-mipi-csi2.0: mipi_csis_s_stream: 0, state: 0x0
mxc-mipi-csi2.0: Frame End events: 8
mxc-mipi-csi2.0: Frame Start events: 8
mxc-mipi-csi2.0: Non-image data after odd frame events: 0
mxc-mipi-csi2.0: Non-image data before odd frame events: 0
mxc-mipi-csi2.0: Non-image data after even frame events: 0
mxc-mipi-csi2.0: Non-image data before even frame events: 0
mxc-mipi-csi2.0: Unknown Error events: 0
mxc-mipi-csi2.0: CRC Error events: 0
mxc-mipi-csi2.0: ECC Error events: 1
mxc-mipi-csi2.0: FIFO Overflow Error events: 0
mxc-mipi-csi2.0: Lost Frame End Error events: 0
mxc-mipi-csi2.0: Lost Frame Start Error events: 1342
mxc-mipi-csi2.0: SOT Error events: 1

joanxie · ‎03-31-2024

you said your IMX412 can work with 2 lane, but why the dump error message for 2 lane settings? and pls check the register Gasket 0 Control Register (GASKET_0_CTRL), this register shouldn't be 0, you should set correct format for your camera, so pls check it