iMX8MP dual ISP capture hangs depending on the CPU grades and revisions

[khang_letruong]

Hi Community,

Following my previous color issue with dual ISP use-case of which root cause was identified in the BSP, I am now struggling with a new issue : when capturing dual ISP outputs in parallel (from /dev/video0 and /dev/video1 in our system), it hangs at below select() call (for /dev/video0 mostly) in the isp-imx-4.2.2.11.0/appshell/v4l_drm_test/video_test.cpp file :

    while (1) {
        fd_set fds;
        FD_ZERO (&fds);
        FD_SET (camDevice, &fds);
        select (camDevice + 1, &fds, NULL, NULL, NULL);
        if (ioctl(camDevice, VIDIOC_DQBUF, &buffer) < 0) {
            ALOGE("VIDIOC_DQBUF: %s", strerror(errno));
        } else {
            ALOGI("VIDIOC_DQBUF success");

Another important observation is that this issue happened constantly on the EVK that I received from Early Access / Beta Program (thus commercial-grade rev. A0 I guess), a little bit less constantly on our 1st production lot with commercial-grade rev. A1 CPU, rarely happens on our 2nd production lot with the industrial-grade CPU. We was developing on this industrial-grade CPU thus unfortunately we recognized the issue quite late with the pre-release running on  commercial-grade rev. A1 CPU based-boards at client side, so it is very critical.

Have you ever experienced the above issue knowing that we are using isp-imx 4.2.2.11.0 align with BSP-5.4.70-2.3.2 (for u-boot, kernel and isp-vvcam, ...) and this is very critical for us in deciding to fix it or moving to newer BSP version that would require huge effort ?

Thanks in advance and look forward to hearing your opinion/advice.

Khang

[khang_letruong]

Hi again,

I enabled the MIPI-CSI2 registers dumping (dump_csis_regs()) and when the hang/timeout happened, it gave :

[   81.812095] mxc-mipi-csi2.0: mipi_csis_s_stream: 1, state: 0x0
[   81.812127] mxc-mipi-csi2.0: fmt: 0x3007, 1440 x 1080
[   81.825286] mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
[   81.825290] mxc-mipi-csi2.0:         CSIS_VERSION[0]: 0x03060301
[   81.825292] mxc-mipi-csi2.0:        CSIS_CMN_CTRL[4]: 0x00004805
[   81.825295] mxc-mipi-csi2.0:        CSIS_CLK_CTRL[8]: 0x000f0000
[   81.825297] mxc-mipi-csi2.0:          CSIS_INTMSK[10]: 0x0fffff1f
[   81.825302] mxc-mipi-csi2.0:          CSIS_INTSRC[14]: 0x00000000
[   81.825304] mxc-mipi-csi2.0:      CSIS_DPHYSTATUS[20]: 0x000000e2
[   81.825309] mxc-mipi-csi2.0:        CSIS_DPHYCTRL[24]: 0x0d800003
[   81.825311] mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_L[30]: 0x000001f4
[   81.825313] mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_H[34]: 0x00000000
[   81.825318] mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_L[38]: 0x00000000
[   81.825323] mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_H[3c]: 0x00000000
[   81.825325] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH0[40]: 0x000000ac
[   81.825327] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH1[50]: 0x000008fd
[   81.825332] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH2[60]: 0x000008fe
[   81.825334] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH3[70]: 0x000008ff
[   81.825339] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH0[44]: 0x043805a0
[   81.825341] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH1[54]: 0x80008000
[   81.825346] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH2[64]: 0x80008000
[   81.825350] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH3[74]: 0x80008000
[   81.825353] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH0[48]: 0x00000000
[   81.825355] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH1[58]: 0x00000000
[   81.825359] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH2[68]: 0x00000000
[   81.825362] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH3[78]: 0x00000000
[   81.825366] mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
[   81.825369] mxc-mipi-csi2.0:    GPR_GASKET_0_CTRL[60]: 0xffff8000
[   81.825371] mxc-mipi-csi2.0:   GPR_GASKET_0_HSIZE[64]: 0xffff8000
[   81.825373] mxc-mipi-csi2.0:   GPR_GASKET_0_VSIZE[68]: 0xffff8000

While in normal streaming :

[   43.033355] mxc-mipi-csi2.0: mipi_csis_s_stream: 1, state: 0x0
[   43.033387] mxc-mipi-csi2.0: fmt: 0x3007, 1440 x 1080
[   43.048990] mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
[   43.048999] mxc-mipi-csi2.0:         CSIS_VERSION[0]: 0x03060301
[   43.049006] mxc-mipi-csi2.0:        CSIS_CMN_CTRL[4]: 0x00004805
[   43.049009] mxc-mipi-csi2.0:        CSIS_CLK_CTRL[8]: 0x000f0000
[   43.049014] mxc-mipi-csi2.0:          CSIS_INTMSK[10]: 0x0fffff1f
[   43.049017] mxc-mipi-csi2.0:          CSIS_INTSRC[14]: 0x00000000
[   43.049020] mxc-mipi-csi2.0:      CSIS_DPHYSTATUS[20]: 0x000000f1
[   43.049023] mxc-mipi-csi2.0:        CSIS_DPHYCTRL[24]: 0x0d800003
[   43.049027] mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_L[30]: 0x000001f4
[   43.049030] mxc-mipi-csi2.0:     CSIS_DPHYBCTRL_H[34]: 0x00000000
[   43.049033] mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_L[38]: 0x00000000
[   43.049038] mxc-mipi-csi2.0:     CSIS_DPHYSCTRL_H[3c]: 0x00000000
[   43.049041] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH0[40]: 0x000000ac
[   43.049047] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH1[50]: 0x000008fd
[   43.049052] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH2[60]: 0x000008fe
[   43.049055] mxc-mipi-csi2.0:   CSIS_ISPCONFIG_CH3[70]: 0x000008ff
[   43.049061] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH0[44]: 0x043805a0
[   43.049064] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH1[54]: 0x80008000
[   43.049069] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH2[64]: 0x80008000
[   43.049074] mxc-mipi-csi2.0:    CSIS_ISPRESOL_CH3[74]: 0x80008000
[   43.049077] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH0[48]: 0x00000000
[   43.049083] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH1[58]: 0x00000000
[   43.049088] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH2[68]: 0x00000000
[   43.049091] mxc-mipi-csi2.0:     CSIS_ISPSYNC_CH3[78]: 0x00000000
[   43.049096] mxc-mipi-csi2.0: --- mipi_csis_s_stream ---
[   43.049099] mxc-mipi-csi2.0:    GPR_GASKET_0_CTRL[60]: 0xffff8000
[   43.049102] mxc-mipi-csi2.0:   GPR_GASKET_0_HSIZE[64]: 0xffff8000
[   43.049108] mxc-mipi-csi2.0:   GPR_GASKET_0_VSIZE[68]: 0xffff8000

The difference I found from above was CSIS_DPHYSTATUS[20] = 0x000000e2 when the issue happened, otherwise CSIS_DPHYSTATUS[20] = 0x000000f1 for normal streaming. And the change seems to relate to the ULPS state :

Best regards,
Khang

[TerryBarnaby1]

Note as far as I understand (can't find any authoritative detailed docs generally available on either CSI2 or the NXP IMX8MP CSI2/ISI/ISP implementation), the CSI2 clock can be operated in a constant clock mode or in a lower power clock on/off mode where it goes from a low power non clocking state to a high speed clock state only when data is being sent (ie not during vertical/horizontal sync and general dead time).

The NXP CSI2's CSIS_DPHYSTATUS register will change its data lane and clock status bits live as the stream runs. So you only take a particular sample and need to repeat reading the register to see what states it goes through.

In my case it looks like the TP2855 is sending its clock in constant mode. When running, if I sample the CSIS_DPHYSTATUS every 1ms and list the different values obtained when the system is showing video (I have a simple 'C' program), I see only 0x000000F0 and 0x00000000. So the clock is not seen to enter the stopped or low power states but the data pairs are going from HS live to stopped. When my system is not showing video (when it should be) I see only 0x000000F1 and 0x00000001. So the clock is only in the stopped state but data pairs are running fine.

The HSSETTLE and CLKSETTLECTL as set by the DTB's mipi_csi_[01] csis-hs-settle and csis-clk-settle params appear to set the hardware to be able to determine the LP11 to LP00 state change that occurs when a clock or data pair transitions from a low power state to high power high speed mode. I don't know what these settings should be set to for the IMX8MP (NXP where is this info ?), I saw info for the IMX7 and there was a table for these values based on the high speed clock rate in use and I have been using that.

In your case, as a complete random comment from just two samples, I notice the clock appears to be going from stopped to running modes so is probably a low power/high power clock. But I note that in your problem state not all of the data pairs enter the stopped state at least at the time of the sample.

Note I am probably leading you down the garden path on this one, as my problem could easily be different to yours.

[TerryBarnaby1]

I think I have worked around my issue. The TP2855 appears to send a constant CSI2 clock. If when the video is not working I tell the TP2855 to set the clock into a stop state and then run state again the video then runs fine. I am guessing here but what I suspect is:

1. When gstreamer/kernel initialises the video stream, the TP2855 is being initialised first and starts the CSI2 clock with appropriate LP00 -> LP11 -> LP00 transition once its PLL has become stable. However I suspect, depending on various timings the CSI2 interface may be initialised a bit later sometimes and the Samsung CSI2 hardware does not notices the LP00 -> LP11 -> LP00 transition as a clock start transition.

2. The Samsung CSI2 IP block hardware appears to require the LP00 -> LP11 -> LP00 transition to determine a high speed clock is present which is ok when you are using a CSI2 source that provides a low-power/high-power switching clock source but can be an issue if the CSI2 has a constant high speed clock as in my case.

NXP can you confirm this please and if there is a mode for the Samsung CSI2 IP block to work well with constant CSI2 clock sources ?

I will look at V4L API functions for the front end driver that can be used to "start the video stream" after the hardware pipeline is up and running where I can get the TP2855 to restart its CSI2 clock  or alternatively add a TP2855 call that the user level software can use if no video stream is present. Or maybe get a vertical sync interrupt from the TP2855 to instigate a clock restart on each frame.

I may have hijacked your thread on this as my issue is likely to be different from yours, but I suggest you check that the CSI2 CSIS_DPHYSTATUS register is indicating that all data pairs and clock pair are transitioning from Stopped to running states and none are stuck in the stopped state. At least that should indicate that the front-end to CSI2 data stream appears to be working when your video stream has locked up.

[khang_letruong]

Hi @TerryBarnaby1 ,

Thanks for sharing your experience. It turns out that the problem on my EVK was due to bad miniSAS cable. I changed the cable and it worked much better on that EVK board. I'm still investigating the issue on the 2nd production lot with commercial-grade rev.A1 CPUs.

[TerryBarnaby1]

1. Yes, each of the 3 boards is the same hardware from the same small production run.

2. I only have one pipeline (as IMX8MP does not support CSI2 virtual channels), but this is configured for either 1920x1080p25 or PAL 720x576i25 analogue cameras and then started. As far as I know the hardware pipeline is: TP2855 -> CSI2 -> ISP -> MEM. Software side driving this is gstreamer.

[TerryBarnaby1]

On 1. Yes the behaviours are slightly different in that one board works fine, the other two have intermittent video stream startup issues. I assume there is a PLL locking issue and/or timeout issues somewhere. The CSI2 hardware D-PHY front end block appears to use some Samsung IP ?

[TerryBarnaby1]

I'm not sure if we are seeing the same issue.

We have a custom IMX8MP board that uses a TP2855 analogue video to CSI2 video interface chip feeding one CSI2 video stream into the IMX8MP. We have both 1920x1080p25 and PAL 720x576i25 camera feeds. We have three prototype boards all with the same hardware. On one board the video is displayed from both camera types fine.

On the other boards, the video fails to start between 50% and 90% of the time, especially on the PAL video. One board is worse than the other. Once the video is started though it is fine.

I am trying to debug this at the moment. What I see is the CSI2 register MIPI_CSIx_DPHY_STATUS has a value 0x000000f1 or 0x00000001 when this locked up video occurs. From the documentation Bit 0 is set to 1 when "Clock lane is in Stop State". From what I can see the TP2855 is sending the same CSI2 clock in both of these situations.

My current guess is that the CSI2 DPHY has not synchronised/noticed the clock within some sort of timeout period and the stop state is being set.

I am using a NXP hardknott Yocto Linux with the 5.10.72 lf-5.10.y kernel branch from git://source.codeaurora.org/external/imx/linux-imx.git.

NXP any ideas on this and how to fix it ?

[khang_letruong]

Hi @TerryBarnaby1 ,

1. Do you mean for the same hardware, their behaviors are different ?

2. Do your dual pipelines (one for 1920x1080p25 and other for PAL 720x576i25) include the ISP blocks or not ?

Best regards,

Khang