ISI_1 got wrong v4l2 formats by mxc_isi_cap_s_fmt_mplane

[jerry_liu]

Hello,

We use our camera(AR0521+AP1302) on imx8M Plus.

we can put the camera on CSI0 (/dev/video0), and we use Gstreamer to preview images on screen.

root@edm-g-imx8mp:~# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=2592,height=1944 ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 7895.349602] bypass csc
[ 7895.352017] input fmt YUV4
[ 7895.355009] output fmt YUYV

However, we use the same camera on CSI1 (/dev/video1), we found it had the wrong format (AR24) and make the preview image is also wrong.

root@edm-g-imx8mp:~# gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw,width=2592,height=1944 ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 8205.795508] input fmt YUV4
[ 8205.798484] output fmt AR24

Then we also tried to force set the format to YUY2.

It can work on CS0 (/dev/video0) but it didn't work on CSI1(/dev/video1):

root@edm-g-imx8mp:~# gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=2592,height=1944 ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../../../../git/libs/gst/base/gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.001685550
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

We got the device info by gst-device-monitor-1.0, and we found there are some lost formats on CSI1 (/dev/video1).

root@edm-g-imx8mp:~# gst-device-monitor-1.0
Probing devices...

Device found:

        name  : Monitor of Dummy Output
        class : Audio/Source
        caps  : audio/x-raw, format=(string){ S16LE, S16BE, F32LE, F32BE, S32LE, S32BE, S24LE, S24BE, S24_32LE, S24_32BE, U8 }, layout=(string)interleaved, rate=(int)[ 1, 384000 ], channels=(int)[ 1, 32 ];
                audio/x-alaw, rate=(int)[ 1, 384000 ], channels=(int)[ 1, 32 ];
                audio/x-mulaw, rate=(int)[ 1, 384000 ], channels=(int)[ 1, 32 ];
        properties:
                device.description = "Monitor\ of\ Dummy\ Output"
                device.class = monitor
                device.icon_name = audio-input-microphone
                is-default = true
        gst-launch-1.0 pulsesrc device=auto_null.monitor ! ...


Device found:

        name  : Dummy Output
        class : Audio/Sink
        caps  : audio/x-raw, format=(string){ S16LE, S16BE, F32LE, F32BE, S32LE, S32BE, S24LE, S24BE, S24_32LE, S24_32BE, U8 }, layout=(string)interleaved, rate=(int)[ 1, 384000 ], channels=(int)[ 1, 32 ];
                audio/x-alaw, rate=(int)[ 1, 384000 ], channels=(int)[ 1, 32 ];
                audio/x-mulaw, rate=(int)[ 1, 384000 ], channels=(int)[ 1, 32 ];
        properties:
                device.description = "Dummy\ Output"
                device.class = abstract
                device.icon_name = audio-card
                is-default = true
        gst-launch-1.0 ... ! pulsesink device=auto_null


Device found:

        name  : mxc-isi-cap
        class : Video/Source
        caps  : video/x-raw, format=(string)YUY2, width=(int)2600, height=(int)1952, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)2592, height=(int)1944, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)2560, height=(int)1440, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1200, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1600, height=(int)1200, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)800, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1024, height=(int)768, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)800, height=(int)600, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)720, height=(int)480, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)640, height=(int)400, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)640, height=(int)360, framerate=(fraction)30/1;
                video/x-raw, format=(string)BGRA, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
                video/x-raw, format=(string)BGRx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
                video/x-raw, format=(string)BGR, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
                video/x-raw, format=(string)RGB, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
                video/x-raw, format=(string)NV12, width=(int)2600, height=(int)1952, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)2592, height=(int)1944, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)2560, height=(int)1440, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1200, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1600, height=(int)1200, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1280, height=(int)800, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1280, height=(int)720, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1024, height=(int)768, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)800, height=(int)600, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)720, height=(int)480, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)640, height=(int)400, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)640, height=(int)360, framerate=(fraction)30/1;
                video/x-raw, format=(string)RGB16, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
        properties:
                udev-probed = true
                device.bus_path = platform-32c00000.bus:camera
                sysfs.path = "/sys/devices/platform/soc\@0/32c00000.bus/32c00000.bus:camera/video4linux/video0"
                device.subsystem = video4linux
                device.product.name = mxc-isi-cap
                device.capabilities = :capture:
                device.api = v4l2
                device.path = /dev/video0
                v4l2.device.driver = mxc-isi-cap
                v4l2.device.card = mxc-isi-cap
                v4l2.device.bus_info = platform:32e00000.isi:cap_devic
                v4l2.device.version = 328774 (0x00050446)
                v4l2.device.capabilities = 2216693760 (0x84201000)
                v4l2.device.device_caps = 69210112 (0x04201000)
        gst-launch-1.0 v4l2src ! ...


Device found:

        name  : mxc-isi-cap
        class : Video/Source
        caps  : video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1200, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1600, height=(int)1200, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)800, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)1024, height=(int)768, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)800, height=(int)600, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)720, height=(int)480, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)640, height=(int)400, framerate=(fraction)30/1;
                video/x-raw, format=(string)YUY2, width=(int)640, height=(int)360, framerate=(fraction)30/1;
                video/x-raw, format=(string)BGRA, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
                video/x-raw, format=(string)BGRx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
                video/x-raw, format=(string)BGR, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
                video/x-raw, format=(string)RGB, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
                video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1200, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1600, height=(int)1200, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1280, height=(int)800, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1280, height=(int)720, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)1024, height=(int)768, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)800, height=(int)600, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)720, height=(int)480, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)640, height=(int)400, framerate=(fraction)30/1;
                video/x-raw, format=(string)NV12, width=(int)640, height=(int)360, framerate=(fraction)30/1;
                video/x-raw, format=(string)RGB16, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ];
        properties:
                udev-probed = true
                device.bus_path = platform-32c00000.bus:camera
                sysfs.path = "/sys/devices/platform/soc\@0/32c00000.bus/32c00000.bus:camera/video4linux/video1"
                device.subsystem = video4linux
                device.product.name = mxc-isi-cap
                device.capabilities = :capture:
                device.api = v4l2
                device.path = /dev/video1
                v4l2.device.driver = mxc-isi-cap
                v4l2.device.card = mxc-isi-cap
                v4l2.device.bus_info = platform:32e02000.isi:cap_devic
                v4l2.device.version = 328774 (0x00050446)
                v4l2.device.capabilities = 2216693760 (0x84201000)
                v4l2.device.device_caps = 69210112 (0x04201000)
        gst-launch-1.0 v4l2src device=/dev/video1 ! ...

We traced the source code and we found the format is decided in mxc_isi_cap_s_fmt_mplane.

Finally, we found pix->pixelformat already was wrong at first. And we think it should be decided from V4L2_IOCTL.

static int mxc_isi_cap_s_fmt_mplane(struct file *file, void *priv,
				    struct v4l2_format *f)
{
	struct mxc_isi_cap_dev *isi_cap = video_drvdata(file);
	struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp;
	struct mxc_isi_frame *dst_f = &isi_cap->dst_f;
	struct mxc_isi_fmt *fmt;
	int bpl;
	int i;

	/* Step1: Check format with output support format list.
	 * Step2: Update output frame information.
	 * Step3: Checkout the format whether is supported by remote subdev
	 *	 Step3.1: If Yes, call remote subdev set_fmt.
	 *	 Step3.2: If NO, call remote subdev get_fmt.
	 * Step4: Update input frame information.
	 * Step5: Update mxc isi channel configuration.
	 */

	dev_dbg(&isi_cap->pdev->dev, "%s, fmt=0x%X\n", __func__, pix->pixelformat);
	if (vb2_is_busy(&isi_cap->vb2_q))
		return -EBUSY;

	/* Check out put format */
	for (i = 0; i < ARRAY_SIZE(mxc_isi_out_formats); i++) {
		fmt = &mxc_isi_out_formats[i];
		if (pix && fmt->fourcc == pix->pixelformat)
			break;
	}

	if (i >= ARRAY_SIZE(mxc_isi_out_formats)) {
		dev_dbg(&isi_cap->pdev->dev,
			"format(%.4s) is not support!\n", (char *)&pix->pixelformat);
		return -EINVAL;
	}

	/* update out put frame size and formate */
	if (pix->height <= 0 || pix->width <= 0)
		return -EINVAL;

	dst_f->fmt = fmt;
	dst_f->height = pix->height;
	dst_f->width = pix->width;

	pix->num_planes = fmt->memplanes;

	for (i = 0; i < pix->num_planes; i++) {
		bpl = pix->plane_fmt[i].bytesperline;

		if ((bpl == 0) || (bpl / (fmt->depth[i] >> 3)) < pix->width)
			pix->plane_fmt[i].bytesperline =
					(pix->width * fmt->depth[i]) >> 3;

		if (pix->plane_fmt[i].sizeimage == 0) {
			if ((i == 1) && (pix->pixelformat == V4L2_PIX_FMT_NV12))
				pix->plane_fmt[i].sizeimage =
				  (pix->width * (pix->height >> 1) * fmt->depth[i] >> 3);
			else
				pix->plane_fmt[i].sizeimage =
					(pix->width * pix->height * fmt->depth[i] >> 3);
		}
	}

	if (pix->num_planes > 1) {
		for (i = 0; i < pix->num_planes; i++) {
			dst_f->bytesperline[i] = pix->plane_fmt[i].bytesperline;
			dst_f->sizeimage[i]    = pix->plane_fmt[i].sizeimage;
		}
	} else {
		dst_f->bytesperline[0] = dst_f->width * dst_f->fmt->depth[0] / 8;
		dst_f->sizeimage[0]    = dst_f->height * dst_f->bytesperline[0];
	}

	memcpy(&isi_cap->pix, pix, sizeof(*pix));
	set_frame_bounds(dst_f, pix->width, pix->height);

	return 0;
}

We don't know why CSI0 can work but CSI1 can't, and why they got different formats while using the same camera.

2021.12.20

In our driver, I set the media format to MEDIA_BUS_FMT_YUYV8_1X16 and it will happen this bug on CSI1.

However, if I set the media format to MEDIA_BUS_FMT_UYVY8_2X8 and it won't happen on CSI1.

This solution can resolve this bug, but I don't still know why that happen even if CSI0 and CSI1 used the same driver.

This bug always happens while previewing the above 2K images but the below 2K doesn't. I think it's related to ISI. Is the problem of ISI_1?

[jerry_liu]

Hello,

the bug happened with kernel version is 5.4.70 and then I found the bug didn't happen if I changed kernel to 5.10.72.

I need to try to fix the bug by comparing the 5.10.72 version kernel.

[joanxie]

this AN is about ISI:

https://www.nxp.com/docs/en/application-note/AN13430.pdf

[joanxie]

in the 5.4.70, for the isi_1, could you read the IMG_CTRL register? what data do you get? and what's the difference between dual isi when you use the command "v4l2-ctl --list-formats-ext -d /dev/videox" ?

I compare with 5.4.70 and 5.10, the i.mx8mp.dtsi is different, and it seems these isi , one isi.m2m and another one is isi.cap

[jerry_liu · Accepted solution]

Hello,

I found isi_1 doesn't support above 2K resolution according to Reference Manual.

Processing channels are capable of handling up to 4K image resolution. However, the line buffer storage in one channel is enough for 2K image resolution. In order to be able to process and store a 4K image, line buffer from adjacent channel needs to be chained with the current channel's line buffer. The very last channel's line buffer cannot be concatenated with other channels, hence 4K resolution cannot be supported on that channel.

So it's a ridiculous problem because I wanted to play 2K images on ISI_1 so that it happened the unpredictable bugs.