Using the V4L2 API for Camera Sensor

Yes

This guide assumes that the developer has knowledge of the V4L2 API and has worked or is familiar with sensor drivers and their operation within the Linux kernel. This guide does not focus on the details of the sensor driver development that you want to port. It is assumed that you already have an existing driver for your sensor, before making the port.

The version of the ISP's was 6.6.36 Linux BSP. If a different version is used, it is the developer's responsibility to review the API documentation for the corresponding version, since there may be changes that affect what is indicated in this guide.

To port the camera sensor, the following steps must be taken as described in the following sections:

Define sensor attributes and create instances.
ISS Driver and ISP Media Server.
Sensor Calibration Files.
VVCAM Driver Creation.
Device Tree Modifications.

Define Sensor Attributes and Create Instances

The following three steps are already implemented in CamDevice and are included for reference only.

Step 1: Define the sensor attributes in the IsiSensor_s data structure.
Step 2: Define the IsiSensorInstanceConfig_t configuration structure that will be used to create a new sensor instance.
Step 3: Call the IsiCreateSensorIss() function to create a new sensor instance.

ISS Driver and ISP Media Server

Step 0 - Use a driver template as base code: Drivers can be found in $ISP_SOURCES_TOP/units/isi/drv/. For example, the ISP sources, come with the OV4656 and OS08a20 drivers. $ISP_SOURCES_TOP indicates the path of your working directory, where the respective sources are located.

Step 1 - Add your <SENSOR> ISS Driver: Create the driver entry for your sensor in the path $ISP_SOURCES_TOP/units/isi/drv/<SENSOR>/source/<SENSOR>.c. Change all occurrences of the respective sensor name within the code, for instance, OV4656 -> <SENSOR>, respecting capital letters where applicable.

Step 2 - Check the information on the IsiCamDrvConfig_s data structure: Data members defined in this data structure include the sensor ID (CameraDriverID) and the function pointer to the IsiSensor data structure. By using the address of the IsiCamDrvConfig_s structure, the driver can then access the sensor API attached to the function pointer. The following is an example of the structure:

/*****************************************************************************

* Each sensor driver needs to declare this struct for ISI load

*****************************************************************************/

IsiCamDrvConfig_t IsiCamDrvConfig = {

.CameraDriverID = 0x0000,

.pIsiHalQuerySensor = <SENSOR>_IsiHalQuerySensorIss,

.pfIsiGetSensorIss = <SENSOR>_IsiGetSensorIss,

};

Important Note: Modify the CameraDriverID according to the chip ID of your sensor. Apply this change to any Chip ID occurrence within the code.

Step 3 - Check sensor macro definitions: In case there is any macro definition in the ISS Driver code, which involves specific properties of the sensor, you should modify it according to your requirements. For example:

#define <SENSOR>_MIN_GAIN_STEP

(1.0f/16.0f)

Step 4 - Modify ISP Media Server build tools: Changes required in this step include:

Add a CMakeLists.txt file in $ISP_SOURCES_TOP/units/isi/drv/<SENSOR>/ that builds your sensor module.
Modify the CMakeLists.txt located at $ISP_SOURCES_TOP/units/isi/drv/CMakeLists.txt to include and reference your sensor directory.
Modify the $ISP_SOURCES_TOP/appshell/ and $ISP_SOURCES_TOP/mediacontrol/ build tools, since by default they refer to the construction of a particular sensor, for example, the OV4656, so it is necessary to change the name of the corresponding sensor.
Modify the $ISP_SOURCES_TOP/build-all-isp.sh script to reference the sensor modules and generate the corresponding binaries when building the ISP media server instance.

Step 5 - ISP Media Server run script: You need to add the operation modes defined for your sensor in the script. Each operating mode is associated with an order (mode 0, mode 1 ... mode N), a name used to execute the command in the terminal (e.g <sensor>_custom_mode_1), a resolution, and a specific calibration file for the sensor. The script is located at $ISP_SOURCES_TOP/imx/run.sh .

Step 6 - Sensor<X> config: At $ISP_SOURCES_TOP/units/isi/drv/ you can find the files to configure each sensor entry to the ISP, called Sensor0_Entry.cfg and Sensor1_Entry.cfg. There, the associated calibration files are indicated for each sensor operating mode, including the calibration files in XML format and the Dewarp Unit configuration files in JSON format. In addition, the .drv file generated for your sensor is referenced, creating the association between the respective /dev/video<X> node and the sensor driver module outputted from the ISP Media Server. In case you are using only one ISP channel, just modify Sensor0_Entry.cfg. In case you require both instances of the ISP, you will need to modify both files.

Sensor Calibration Files

It is a requirement for using the ISP, to have a calibration file in XML format, specific to the sensor you are using and according to the resolution and working mode. To obtain the calibration files in XML format, there are 3 options:

Use the NXP ISP tuning tool for this you will need to ask for access or sign a NDA document.
Pay NXP professional services to do the tune.
Pay a third-party vendor to do the tune

VVCAM Driver Creation

The changes indicated below are based on the assumption that there is a functional sensor driver in its base form, and that it is compatible with the V4L2 API. From now on we focus on applying the changes suggested in the NXP documentation, specifically to establish the communication of the VVCAM Driver (kernel side) and the ISI Layer.

Step 0 - Create the sensor driver entry: Developers must add the driver code to the file located at $ISP_SOURCES_TOP/vvcam/v4l2/sensor/<sensor>/<sensor>_xxxx.c, along with a Makefile for the sensor driver module. In the same way, as indicated in the ISS Driver section, you can refer to one of the sample drivers that are included as part of the ISP sources, to review details about the implementation of the driver and the structure of the required Makefile.

Step 1 - Add the VVCAM mode info data structure array: This array stores all the supported modes information for your sensor. The ISI layer can get all the modes with the VVSENSORIOC_QUERY command. The following is an example of the structure, please fill in the information using the attributes of your sensor and the modes it supports.

#include "vvsensor.h"

.

static struct vvcam_mode_info_s <sensor>_mode_info[] = {

{

.index = 0,

.width = ... ,

.height = ... ,

.hdr_mode = ... ,

.bit_width = ... ,

.data_compress.enable = ... ,

.bayer_pattern = ... ,

.ae_info = {

.

},

.mipi_info = {

.mipi_lane = ... ,

},

{

.index = 1,

.

},

};

Step 2 - Define sensor client to i2c : Define the client_to_sensor macro (in case you don't have any already) and check the segments of the driver code that require this macro.

#define client_to_<sensor>(client)\

container_of(i2c_get_clientdata(client), struct <sensor>, subdev)

Step 3 - Define the V4L2-subdev IOCTL function: Define and implement the <sensor>_priv_ioctl, which is used to receive the commands and parameters passed down by the user space through ioctl() and control the sensor.

long <sensor>_priv_ioctl(struct v4l2_subdev *subdev, unsigned int cmd, void *arg)

{

struct i2c_client *client = v4l2_get_subdevdata(subdev);

struct <sensor> *sensor = client_to_<sensor>(client);

struct vvcam_sccb_data_s reg;

uint32_t value = 0;

long ret = 0;

if(!sensor){

return -EINVAL;

}

switch (cmd) {

case VVSENSORIOC_G_CLK: {