This guide assumes that the developer has knowledge of the V4L2 API and has worked or is familiar with sensor drivers and their operation within the Linux kernel. This guide does not focus on the details of the sensor driver development that you want to port. It is assumed that you already have an existing driver for your sensor, before making the port.
The version of the ISP's was 6.6.36 Linux BSP. If a different version is used, it is the developer's responsibility to review the API documentation for the corresponding version, since there may be changes that affect what is indicated in this guide.
To port the camera sensor, the following steps must be taken as described in the following sections:
Define sensor attributes and create instances.
ISS Driver and ISP Media Server.
Sensor Calibration Files.
VVCAM Driver Creation.
Device Tree Modifications.
Define Sensor Attributes and Create Instances
The following three steps are already implemented in CamDevice and are included for reference only.
Step 1: Define the sensor attributes in the IsiSensor_s data structure.
Step 2: Define the IsiSensorInstanceConfig_t configuration structure that will be used to create a new sensor instance.
Step 3: Call the IsiCreateSensorIss() function to create a new sensor instance.
ISS Driver and ISP Media Server
Step 0 - Use a driver template as base code: Drivers can be found in $ISP_SOURCES_TOP/units/isi/drv/. For example, the ISP sources, come with the OV4656 and OS08a20 drivers. $ISP_SOURCES_TOP indicates the path of your working directory, where the respective sources are located.
Step 1 - Add your <SENSOR> ISS Driver: Create the driver entry for your sensor in the path $ISP_SOURCES_TOP/units/isi/drv/<SENSOR>/source/<SENSOR>.c. Change all occurrences of the respective sensor name within the code, for instance, OV4656 -> <SENSOR>, respecting capital letters where applicable.
Step 2 - Check the information on the IsiCamDrvConfig_s data structure: Data members defined in this data structure include the sensor ID (CameraDriverID) and the function pointer to the IsiSensor data structure. By using the address of the IsiCamDrvConfig_s structure, the driver can then access the sensor API attached to the function pointer. The following is an example of the structure:
/*****************************************************************************
* Each sensor driver needs to declare this struct for ISI load
*****************************************************************************/
IsiCamDrvConfig_t IsiCamDrvConfig = {
.CameraDriverID = 0x0000,
.pIsiHalQuerySensor = <SENSOR>_IsiHalQuerySensorIss,
.pfIsiGetSensorIss = <SENSOR>_IsiGetSensorIss,
};
Important Note: Modify the CameraDriverID according to the chip ID of your sensor. Apply this change to any Chip ID occurrence within the code.
Step 3 - Check sensor macro definitions: In case there is any macro definition in the ISS Driver code, which involves specific properties of the sensor, you should modify it according to your requirements. For example:
#define <SENSOR>_MIN_GAIN_STEP
(1.0f/16.0f)
Step 4 - Modify ISP Media Server build tools: Changes required in this step include:
Add a CMakeLists.txt file in $ISP_SOURCES_TOP/units/isi/drv/<SENSOR>/ that builds your sensor module.
Modify the CMakeLists.txt located at $ISP_SOURCES_TOP/units/isi/drv/CMakeLists.txt to include and reference your sensor directory.
Modify the $ISP_SOURCES_TOP/appshell/ and $ISP_SOURCES_TOP/mediacontrol/ build tools, since by default they refer to the construction of a particular sensor, for example, the OV4656, so it is necessary to change the name of the corresponding sensor.
Modify the $ISP_SOURCES_TOP/build-all-isp.sh script to reference the sensor modules and generate the corresponding binaries when building the ISP media server instance.
Step 5 - ISP Media Server run script: You need to add the operation modes defined for your sensor in the script. Each operating mode is associated with an order (mode 0, mode 1 ... mode N), a name used to execute the command in the terminal (e.g <sensor>_custom_mode_1), a resolution, and a specific calibration file for the sensor. The script is located at $ISP_SOURCES_TOP/imx/run.sh .
Step 6 - Sensor<X> config: At $ISP_SOURCES_TOP/units/isi/drv/ you can find the files to configure each sensor entry to the ISP, called Sensor0_Entry.cfg and Sensor1_Entry.cfg. There, the associated calibration files are indicated for each sensor operating mode, including the calibration files in XML format and the Dewarp Unit configuration files in JSON format. In addition, the .drv file generated for your sensor is referenced, creating the association between the respective /dev/video<X> node and the sensor driver module outputted from the ISP Media Server. In case you are using only one ISP channel, just modify Sensor0_Entry.cfg. In case you require both instances of the ISP, you will need to modify both files.
Sensor Calibration Files
It is a requirement for using the ISP, to have a calibration file in XML format, specific to the sensor you are using and according to the resolution and working mode. To obtain the calibration files in XML format, there are 3 options:
Use the NXP ISP tuning tool for this you will need to ask for access or sign a NDA document.
Pay NXP professional services to do the tune.
Pay a third-party vendor to do the tune
VVCAM Driver Creation
The changes indicated below are based on the assumption that there is a functional sensor driver in its base form, and that it is compatible with the V4L2 API. From now on we focus on applying the changes suggested in the NXP documentation, specifically to establish the communication of the VVCAM Driver (kernel side) and the ISI Layer.
Step 0 - Create the sensor driver entry: Developers must add the driver code to the file located at $ISP_SOURCES_TOP/vvcam/v4l2/sensor/<sensor>/<sensor>_xxxx.c, along with a Makefile for the sensor driver module. In the same way, as indicated in the ISS Driver section, you can refer to one of the sample drivers that are included as part of the ISP sources, to review details about the implementation of the driver and the structure of the required Makefile.
Step 1 - Add the VVCAM mode info data structure array: This array stores all the supported modes information for your sensor. The ISI layer can get all the modes with the VVSENSORIOC_QUERY command. The following is an example of the structure, please fill in the information using the attributes of your sensor and the modes it supports.
#include "vvsensor.h"
.
.
.
static struct vvcam_mode_info_s <sensor>_mode_info[] = {
{
.index = 0,
.width = ... ,
.height = ... ,
.hdr_mode = ... ,
.bit_width = ... ,
.data_compress.enable = ... ,
.bayer_pattern = ... ,
.ae_info = {
.
.
.
},
.mipi_info = {
.mipi_lane = ... ,
},
},
{
.index = 1,
.
.
.
},
};
Step 2 - Define sensor client to i2c : Define the client_to_sensor macro (in case you don't have any already) and check the segments of the driver code that require this macro.
#define client_to_<sensor>(client)\
container_of(i2c_get_clientdata(client), struct <sensor>, subdev)
Step 3 - Define the V4L2-subdev IOCTL function: Define and implement the <sensor>_priv_ioctl, which is used to receive the commands and parameters passed down by the user space through ioctl() and control the sensor.
long <sensor>_priv_ioctl(struct v4l2_subdev *subdev, unsigned int cmd, void *arg)
{
struct i2c_client *client = v4l2_get_subdevdata(subdev);
struct <sensor> *sensor = client_to_<sensor>(client);
struct vvcam_sccb_data_s reg;
uint32_t value = 0;
long ret = 0;
if(!sensor){
return -EINVAL;
}
switch (cmd) {
case VVSENSORIOC_G_CLK: {
ret = custom_implementation();
break;
}
case VIDIOC_QUERYCAP: {
ret = custom_implementation();
break;
}
case VVSENSORIOC_QUERY: {
ret = custom_implementation();
break;
}
case VVSENSORIOC_G_CHIP_ID: {
ret = custom_implementation();
break;
}
case VVSENSORIOC_G_RESERVE_ID: {
ret = custom_implementation();
break;
}
case VVSENSORIOC_G_SENSOR_MODE:{
ret = custom_implementation();
break;
}
case VVSENSORIOC_S_SENSOR_MODE: {
ret = custom_implementation();
break;
}
case VVSENSORIOC_S_STREAM: {
ret = custom_implementation();
break;
}
case VVSENSORIOC_WRITE_REG: {
ret = custom_implementation();
break;
}
case VVSENSORIOC_READ_REG: {
ret = custom_implementation();
break;
}
case VVSENSORIOC_S_EXP: {
ret = custom_implementation();
break;
}
case VVSENSORIOC_S_POWER:
case VVSENSORIOC_S_CLK:
case VVSENSORIOC_RESET:
case VVSENSORIOC_S_FPS:
case VVSENSORIOC_G_FPS:
case VVSENSORIOC_S_LONG_GAIN:
case VVSENSORIOC_S_GAIN:
case VVSENSORIOC_S_VSGAIN:
case VVSENSORIOC_S_LONG_EXP:
case VVSENSORIOC_S_VSEXP:
case VVSENSORIOC_S_WB:
case VVSENSORIOC_S_BLC:
case VVSENSORIOC_G_EXPAND_CURVE:
break;
default:
break;
}
return ret;
}
As you can see in the example, some cases are implemented but others are not. Developers are free to implement the features they consider necessary, as long as a minimum base of operation of the driver is guaranteed (query commands, read and write registers, among others). It is the developer's responsibility to implement each custom function, for each case or scenario that may arise when interacting with the sensor. In addition to what was shown previously, a link must be created to make the ioctl connection with the driver in question. Link your priv_ioctl function on the v4l2_subdev_core_ops struct, as in the example below:
static const struct v4l2_subdev_core_ops <sensor>_core_ops = {
.s_power = v4l2_s_power,
.subscribe_event = v4l2_ctrl_subdev_subscribe_event,
.unsubscribe_event = v4l2_event_subdev_unsubscribe,
// IOCTL link
.ioctl = <sensor>_priv_ioctl,
};
Step 4 - Verify your sensor's private data structure: After performing the modifications suggested, it would be a good practice to double-check your sensor's private data structure properties, in case there is one missing, and also check that the properties are initialized correctly on the driver's probe.
Step 5 - Modify VVCAM V4L2 sensor Makefile : At $ISP_SOURCES_TOP/vvcam/v4l2/sensor/Makefile, include your sensor object as follows:
...
obj-m += <sensor>/
...
Important Note: There is a very common issue that appears when working with camera sensor drivers in i.MX8MP platforms. The kernel log message shows something similar to the following:
mxc-mipi-csi2.<X>: is_entity_link_setup, No remote pad found!
The link setup callback is required by the Media Controller when performing the linking process of the media entities involved in the capture process of the camera. Normally, this callback is triggered by the imx8-media-dev driver included as part of the Kernel sources. To make sure that the problem is not related to your sensor driver, verify the link setup callback is already created in the code, and if is not, you can add the following template:
/* Function needed by i.MX8MP */
static int <sensor>_camera_link_setup(struct media_entity *entity,
const struct media_pad *local,
const struct media_pad *remote, u32 flags)
{
/* Return always zero */
return 0;
}
/* Add the link setup callback to the media entity operations struct */
static const struct media_entity_operations <sensor>_camera_subdev_media_ops = {
.link_setup = <sensor>_camera_link_setup,
};
/* Verify the initialization process of the media entity ops in the sensor driver's probe function*/
static int <sensor>_probe(struct i2c_client *client, ...)
{
/* Initialize subdev */
sd = &<sensor>->subdev;
sd->dev = &client->dev;
<sensor>->subdev.internal_ops = ...
<sensor>->subdev.flags |= ...
<sensor->subdev.entity.function = ...
/* Entity ops initialization */
<sensor->subdev.entity.ops = &<sensor>_camera_subdev_media_ops;
}
In most cases, adding the link setup function will solve the media controller issue, or at least it discards problems on the driver side.
Device Tree Modifications
On the Device Tree side, it is necessary to enable the ISP channels that will be used. Likewise, it is necessary to disable the ISI channels, which are normally the ones that connect to the MIPI_CSI2 ports to extract raw data from the sensor (in case the ISP is not used). A MIPI_CSI2 port can be mapped to either an ISI channel or an ISP channel, but not both simultaneously. In this guide, we focus on using the ISP, so any other custom configuration that you want to implement may vary from what is shown. In the code below, ISP channel 0 is enabled, and the connection is made to the port where the sensor is connected (mipi_csi_0).
&mipi_csi_0 {
status = "okay";
port@0 {
// Example endpoint to <sensor>_ep
mipi0_sensor_ep: endpoint@1 {
remote-endpoint = <&<sensor>_ep>;
};
};
};
&cameradev {
status = "okay";
};
&isi_0 {
status = "disabled";
};
&isi_1 {
status = "disabled";
};
&isp_0 {
status = "okay";
};
&isp_1 {
status = "disabled";
};
&dewarp {
status = "okay";
};
What is shown above does not represent a complete device tree file, is only a general skeleton of the points you should pay attention to when working with ISP channels. For simplicity, we omitted all the attributes that are normally defined when working with camera sensor drivers and their respective configurations in the i2c port of the hardware.
Note: Due to hardware restrictions when using ISP channels, it is recommended to use the isp_0 channel, when working with only one sensor. In case you need to use two sensors, you can enable both channels, taking into account the limitations regarding the output resolutions and the clock frequency when both channels are working simultaneously. What is not recommended is to use the isp_1 channel when working with a single sensor.
References
ISP Independent Sensor Interface (ISI) API reference, I.MX8M Plus Camera Sensor Porting User guide: https://www.nxp.com/webapp/Download?colCode=IMX8MPCSPUG
Sensor Calibration tool: https://www.nxp.com/webapp/Download?colCode=AN13565
i.MX8M Plus reference manual: https://www.nxp.com/webapp/Download?colCode=IMX8MPRM
View full article
