Mipi-csi2 capture misalignment using ISP on imx8mp

[oliverlewis]

We've recently migrated from capturing from an ISI based pipeline to using the ISP. The aim is to perform the debayer in h/w instead of software.

The ISI version of the driver works and can support two instances of our sensor on two separate 4-line mipi-csi2 interfaces. The resolution and bit depths aren't scary either (1920x1200, 8 or 10-bit).

The ISP version of the driver is the same as the ISI version apart from opening up the additional ioctl's required by the vvcam service. The sensor interface driver/lib which links vvcam also appears to start successfully and runs successfully. Adapted xml and json config files seem correct too.

The isp_media_server works for one or two instances of our sensor, installing all the correct system drivers and sensor drivers (imx8_media_dev, our driver, vvcam_video, vvcam_dwp, and vvcam_isp). It reports no errors and installs a /dev/videoX instance per sensor correctly.

Using gstreamer or the video_test tool work too and kick off the correct sensor in the correct mode. However, what we get out is wrong.

Attached is a capture from video_test that was output to a monitor. It looks like the mipi setup isn't correct when it's done indirectly through isp_media_server instead of using the ISI/device tree approach. It appears to be missing SOF headers and misaligns the capture. 

Here are the versions we use:

• Varascite SOM based imx8mp
• Kirkstone yocto build
• Kernel v5.15.60
• isp-imx-4.2.2.18.0
• kernel-module-isp-vvcam-4.2.2.18.0

Any pointers to where these settings are configured would be great. My assumption was the device tree, though apart from disabling ISI and enabling ISP, this has not changed.

Many thanks.

- Oliver

[oliverlewis]

Sorry, attachment didn't take when topic was posted....

[joanxie]

what camera do you use? pls offer the test log with me, did you try other format? any difference?

[oliverlewis]

@joanxie, thanks for getting back. The sensor is an ar0234, where the first sensor on isp0 is the monochrome variant and the sensor on isp1 is a colour (bayer) variant. There is no difference in the way the sensors are driven under the hood.

The logs are a little bit useless as nothing is failing. But attached is the output from the isp_media_server anyway. Not sure what other logs would be interesting.

There is no real change between the two sensors (colour or mono) and I wouldn't expect that as the data captured and configuration would be the same for both sensors. There is a difference if I change the capture resolution from 10-bit to 8-bit. There is less loss (only 3 repeated images inside the one, instead of about 5), but the SOF is still off and the same misalignment exists. See attached image.

Sorry about the poor quality captures, I am literally taking a photo of a screen with my phone. But I think it shows the problem clear enough.

[joanxie]

did you capture from the dual camera in the same time? and you got the same issue from the dual camera? how about testing one by one? let me remind, if you need to use dual ISP int he same time, only can support up to 1080p resolution, so I suggest that you can test them one by one, we have verified ar0144, maybe you can refer to that, to check if you set correct, check the vvcam driver to set correct 

https://github.com/nxp-imx-support/meta-imx8mp-isp-ar0144

you also can refer to the AN for how to set monochrome

[oliverlewis]

Thanks for the heads up, I wasn't aware that the ISP was limited to 1080p. That is not really an issue as I can run the mono camera through the ISI and just run the colour one through ISP. 

However, I have been running just one camera at a time for now.

I will look at the ar0144 driver to see if there are any obvious differences. Do you know what version of kernel/vvcam/isp the ar0144 driver expecting?

[joanxie]

refer to your description, you need double check your camera driver, the ar0144 SW package is based on 6.1.22