[iMX8MP][ISI] How to properly implement V4L2 camera controls of gain and exposure?

[khang_letruong]

Dear experts,

I'm currently developing a custom camera driver and I would like to handle gain and exposure settings. The camera driver itself supports V4L2_CID_GAIN & V4L2_CID_EXPOSURE.

However, it seems that these setting controls cannot be requested from user-space (via v4l2-ctrl then ioctl) to the kernel due to the limitation of the capture block's driver (which is the ISI since I'm using iMX8M Plus EVK)
<kernel-bsp>/drivers/staging/media/imx/imx8-isi-cap.c :

static int mxc_isi_s_ctrl(struct v4l2_ctrl *ctrl)
{
	struct mxc_isi_cap_dev *isi_cap = ctrl_to_isi_cap(ctrl);
	struct mxc_isi_dev *mxc_isi = mxc_isi_get_hostdata(isi_cap->pdev);
	unsigned long flags;
	dev_dbg(&isi_cap->pdev->dev, "%s\n", __func__);
	if (ctrl->flags & V4L2_CTRL_FLAG_INACTIVE)
		return 0;
	spin_lock_irqsave(&mxc_isi->slock, flags);
	switch (ctrl->id) {
	case V4L2_CID_HFLIP:
		if (ctrl->val < 0)
			return -EINVAL;
		mxc_isi->hflip = (ctrl->val > 0) ? 1 : 0;
		break;
	case V4L2_CID_VFLIP:
		if (ctrl->val < 0)
			return -EINVAL;
		mxc_isi->vflip = (ctrl->val > 0) ? 1 : 0;
		break;
	case V4L2_CID_ALPHA_COMPONENT:
		if (ctrl->val < 0 || ctrl->val > 255)
			return -EINVAL;
		mxc_isi->alpha = ctrl->val;
		mxc_isi->alphaen = 1;
		break;
	default:
		dev_err(&isi_cap->pdev->dev,
			"%s: Not support %d CID\n", __func__, ctrl->id);
		return -EINVAL;
	}
	spin_unlock_irqrestore(&mxc_isi->slock, flags);
	return 0;
}

And above code snippet corresponds with the output of `v4l2-ctl --list-ctrls` command :

# v4l2-ctl --list-ctrls
User Controls
                horizontal_flip 0x00980914 (bool)   : default=0 value=0
                  vertical_flip 0x00980915 (bool)   : default=0 value=0
                alpha_component 0x00980929 (int)    : min=0 max=255 step=1 default=0 value=0

Meanwhile, the capture driver of iMX8M Mini seems to support better V4L2 controls in <kernel-bsp>/drivers/media/platform/mxc/capture/mxc_v4l2_capture.c :

static int mxc_v4l2_s_ctrl(cam_data *cam, struct v4l2_control *c)
{
	int i, ret = 0;
	int tmp_rotation = IPU_ROTATE_NONE;
	struct sensor_data *sensor_data;
	pr_debug("In MVC:mxc_v4l2_s_ctrl\n");
	switch (c->id) {
	case V4L2_CID_HFLIP:
		/* This is done by the IPU */
		if (c->value == 1) {
			if ((cam->rotation != IPU_ROTATE_VERT_FLIP) &&
			    (cam->rotation != IPU_ROTATE_180))
				cam->rotation = IPU_ROTATE_HORIZ_FLIP;
			else
				cam->rotation = IPU_ROTATE_180;
		} else {
			if (cam->rotation == IPU_ROTATE_HORIZ_FLIP)
				cam->rotation = IPU_ROTATE_NONE;
			if (cam->rotation == IPU_ROTATE_180)
				cam->rotation = IPU_ROTATE_VERT_FLIP;
		}
		break;
	case V4L2_CID_VFLIP:
		/* This is done by the IPU */
		if (c->value == 1) {
			if ((cam->rotation != IPU_ROTATE_HORIZ_FLIP) &&
			    (cam->rotation != IPU_ROTATE_180))
				cam->rotation = IPU_ROTATE_VERT_FLIP;
			else
				cam->rotation = IPU_ROTATE_180;
		} else {
			if (cam->rotation == IPU_ROTATE_VERT_FLIP)
				cam->rotation = IPU_ROTATE_NONE;
			if (cam->rotation == IPU_ROTATE_180)
				cam->rotation = IPU_ROTATE_HORIZ_FLIP;
		}
		break;
	case V4L2_CID_MXC_ROT:
	case V4L2_CID_MXC_VF_ROT:
		/* This is done by the IPU */
		switch (c->value) {
		case V4L2_MXC_ROTATE_NONE:
			tmp_rotation = IPU_ROTATE_NONE;
			break;
		case V4L2_MXC_ROTATE_VERT_FLIP:
			tmp_rotation = IPU_ROTATE_VERT_FLIP;
			break;
		case V4L2_MXC_ROTATE_HORIZ_FLIP:
			tmp_rotation = IPU_ROTATE_HORIZ_FLIP;
			break;
		case V4L2_MXC_ROTATE_180:
			tmp_rotation = IPU_ROTATE_180;
			break;
		case V4L2_MXC_ROTATE_90_RIGHT:
			tmp_rotation = IPU_ROTATE_90_RIGHT;
			break;
		case V4L2_MXC_ROTATE_90_RIGHT_VFLIP:
			tmp_rotation = IPU_ROTATE_90_RIGHT_VFLIP;
			break;
		case V4L2_MXC_ROTATE_90_RIGHT_HFLIP:
			tmp_rotation = IPU_ROTATE_90_RIGHT_HFLIP;
			break;
		case V4L2_MXC_ROTATE_90_LEFT:
			tmp_rotation = IPU_ROTATE_90_LEFT;
			break;
		default:
			ret = -EINVAL;
		}
		#ifdef CONFIG_MXC_IPU_PRP_VF_SDC
		if (c->id == V4L2_CID_MXC_VF_ROT)
			cam->vf_rotation = tmp_rotation;
		else
			cam->rotation = tmp_rotation;
		#else
			cam->rotation = tmp_rotation;
		#endif
		break;
	case V4L2_CID_HUE:
		if (cam->sensor) {
			cam->hue = c->value;
			ret = vidioc_int_s_ctrl(cam->sensor, c);
		} else {
			pr_err("ERROR: v4l2 capture: slave not found!\n");
			ret = -ENODEV;
		}
		break;
	case V4L2_CID_CONTRAST:
		if (cam->sensor) {
			cam->contrast = c->value;
			ret = vidioc_int_s_ctrl(cam->sensor, c);
		} else {
			pr_err("ERROR: v4l2 capture: slave not found!\n");
			ret = -ENODEV;
		}
		break;
	case V4L2_CID_BRIGHTNESS:
		if (cam->sensor) {
			cam->bright = c->value;
			ret = vidioc_int_s_ctrl(cam->sensor, c);
		} else {
			pr_err("ERROR: v4l2 capture: slave not found!\n");
			ret = -ENODEV;
		}
		break;
	case V4L2_CID_SATURATION:
		if (cam->sensor) {
			cam->saturation = c->value;
			ret = vidioc_int_s_ctrl(cam->sensor, c);
		} else {
			pr_err("ERROR: v4l2 capture: slave not found!\n");
			ret = -ENODEV;
		}
		break;
	case V4L2_CID_RED_BALANCE:
		if (cam->sensor) {
			cam->red = c->value;
			ret = vidioc_int_s_ctrl(cam->sensor, c);
		} else {
			pr_err("ERROR: v4l2 capture: slave not found!\n");
			ret = -ENODEV;
		}
		break;
	case V4L2_CID_BLUE_BALANCE:
		if (cam->sensor) {
			cam->blue = c->value;
			ret = vidioc_int_s_ctrl(cam->sensor, c);
		} else {
			pr_err("ERROR: v4l2 capture: slave not found!\n");
			ret = -ENODEV;
		}
		break;
	case V4L2_CID_EXPOSURE:
		if (cam->sensor) {
			cam->ae_mode = c->value;
			ret = vidioc_int_s_ctrl(cam->sensor, c);
		} else {
			pr_err("ERROR: v4l2 capture: slave not found!\n");
			ret = -ENODEV;
		}
		break;
	case V4L2_CID_MXC_FLASH:
#ifdef CONFIG_MXC_IPU_V1
		ipu_csi_flash_strobe(true);
#endif
		break;
	case V4L2_CID_MXC_SWITCH_CAM:
		if (cam->sensor == cam->all_sensors[c->value])
			break;
		/* power down other cameraes before enable new one */
		for (i = 0; i < cam->sensor_index; i++) {
			if (i != c->value) {
				vidioc_int_dev_exit(cam->all_sensors[i]);
				vidioc_int_s_power(cam->all_sensors[i], 0);
				if (cam->mclk_on[cam->mclk_source]) {
					ipu_csi_enable_mclk_if(cam->ipu,
							CSI_MCLK_I2C,
							cam->mclk_source,
							false, false);
					cam->mclk_on[cam->mclk_source] =
								false;
				}
			}
		}
		sensor_data = cam->all_sensors[c->value]->priv;
		if (sensor_data->io_init)
			sensor_data->io_init();
		cam->sensor = cam->all_sensors[c->value];
		cam->mclk_source = sensor_data->mclk_source;
		ipu_csi_enable_mclk_if(cam->ipu, CSI_MCLK_I2C,
				       cam->mclk_source, true, true);
		cam->mclk_on[cam->mclk_source] = true;
		vidioc_int_s_power(cam->sensor, 1);
		vidioc_int_dev_init(cam->sensor);
		break;
	default:
		pr_debug("   default case\n");
		ret = -EINVAL;
		break;
	}
	return ret;

Is anybody know if NXP would update the ISI capture driver to handle more V4L2 controls or one need to pass by the ISP for that purpose ?

Thanks in advance and best regards,
Khang

[jimmychan]

Hello,

I got the information from the AE team.

As far as the support for camera controls of gain and exposure should be handled by your Camera Sensor and its corresponding ISP. 

However if you do expect the ISI of our iMX8M Plus to handle these gain and exposure controls, you may have to make these customizations on your own. 

I have attached the sequence flow diagram of v4l2 kernel sequence of how the application sends out its ioctl and queries the capabilities of the ISI and subsequently performs the required sensor specific operations by creating special ioctl to handle frame sizes and intervals of the ov5640 sensor that is connected to our MIPI CSI. 

The entire source code is available as part of the same kernel-source link that you have provided below and you can utilize the attached sequence flow to add the required modification corresponding to your camera sensor module and add the additional IOCtl's that are required to control the gain and exposure level of the sensor, similar to how it is done for ov5460. 

This ISI capture driver is very straight forward and if you want to expand controls, just need to add new control id like V4L2_CID_XXXX, and for the driver implementation, please refer to kernel doc (imx.rst).

If you are using the GA (5.4.70_2.3.0) BSP release, the sequence diagram modules are renamed as follows

• mxc_isi_cap.c -> ./drivers/staging/media/imx/imx8-isi-cap.c
• mxc_mipi_csi2_sam.c -> ./drivers/staging/media/imx/imx8-mipi-csi2-sam.c
• ov5640_mipi_v3.c -> ./drivers/media/platform/mxc/capture/ov5640.c

I hope this information will enable you to expand the ISI capture driver with additional camera controls as you have requested for. 

[malik_cisse]

Hi Jimmy,

From which document is v4l2src_camera_kernel_sequence.jpg taken from?

Also is there a description somewhere how the various drivers (see bellow) interact with each other at kernel load time?

I am porting my own sensor driver and have hard time understanding the various registration mechanisms of the kernel sub-drivers

• drivers/staging/media/imx/imx8-isi-cap.c
• drivers/staging/media/imx/imx8-mipi-csi2-sam.c
• drivers/media/platform/mxc/capture/ov5640.c

Thank you

[khang_letruong]

Thanks @jimmychan for you instruction. I will try to implement this and let you know.

Best regards,

Khang