iMX8MP with 2 cameras using ISP

[Charled]

Hi,

I am trying to use 2 different cameras (in ISP mode) on the EVK board with iMX8MP chipset. I am running Android 11.0.0_1.2.0.

The camera devices are:

• Default Basler provided by NXP
• OV5640, with a 1st porting to ISP mode (because the default BSP only provide the ISI mode)

I am able to use them individually (separately), when they are connected to CSI1.

Now I want to use Basler on CSI1, and OV5640 on CSI2, but then only Basler is seen, and I cannot use OV5640.

Here is the dts I use (based on the default imx8mp-evk-basler.dts):

// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
/*
 * Copyright 2020 NXP
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * as published by the Free Software Foundation; either version 2
 * of the License, or (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 */

#include "imx8mp-evk.dts"

&i2c2 {
	/delete-node/ov5640_mipi@3c;

	basler_camera@36 {
		compatible = "basler,basler-camera-vvcam", "basler-camera-vvcam";
		reg = <0x36>;
		csi_id = <0x00>;
		status = "okay";

		port {
			basler_ep_0: endpoint {
				data-lanes = <1 2 3 4>;
				clock-lanes = <0>;
				link-frequencies = /bits/ 64 <248000000>;
				remote-endpoint = <&mipi_csi0_ep>;
			};
		};
	};
};

&mipi_csi_0 {
	status = "okay";

	port@0 {
		mipi_csi0_ep: endpoint {
			remote-endpoint = <&basler_ep_0>;
			data-lanes = <4>;
			csis-hs-settle = <16>;
		};
	};
};

&ov5640_1 {
	powerdown-gpios = <&gpio2 11 GPIO_ACTIVE_HIGH>;
	reset-gpios = <&gpio1 6 GPIO_ACTIVE_LOW>;
	compatible = "ovti,ov5640_isp";
	csi_id = <1>;

	status = "okay";
};

&mipi_csi_1 {
	status = "okay";
};

&isi_0 {
	status = "disabled";
};

&isi_1 {
	status = "disabled";
};

&isp_0 {
	status = "okay";
};

&isp_1 {
	status = "okay";
};

&dewarp {
	status = "okay";
};

Both camera are detected by the hw and the kernel, and I can see the kernel creating the corresponding devices and their links.

I have used 2 config files Sensor0_Entry.cfg and Sensor1_Entry.cfg in order to allow isp_media_server to find the information for both cameras.

But I cannot use the OV5640 Camera connected on CSI2 (ISP1).

I have also investigated to use only OV5640 on CSI2, without nothing connected on CSI1... without success.

Anyone here who faced the same issue? Any idea how to fix that?

Is there a way to configure only a single camera on CSI2 (and then mapped on ISP1)?

Thanks for your support,

Charles-Edouard

[khang_letruong]

Hi Charles,

In Linux, there's run.sh script (/opt/imx8-isp/bin/) which is part of isp-imx that is in charge of loading .drv, XML, .json, isp_media_server and ISP based sensor drivers. But it supports single camera sensor or dual but identical camera sensors. You can try to modify it to run single ov5640 camera. But I'm not sure if the use-case of 2 different cameras is supported.

Best Regards.

Khang

[Charled]

Thanks @khang_letruong,

Unfortunately, I am using an Android BSP 11.0.0_1.2.0, not Linux. The isp-imx package I have is 4.2.2.6.0.

Moreover, the isp_media_server process is started with a json file as parameter, not an ID indicating if I want to use CAMERA0, CAMERA1, or both.

Which solution do I have with my current BSP and isp-imx package, considering I cannot modify the isp_media_server, because it cannot be compiled and used under Android in this package version?

Thanks,

Best Regards

[joanxie]

why don't you use imx8mp-evk-basler-ov5640.dtb? which is for dual camera basler plus ov5640, ov5640 with built-in ISP uses ISI

[Charled]

Hi @joanxie,

The difficulty is that OV5640 works in ISI mode by default. I need to use 2 camera devices in ISP mode because my devices only support Bayer output format.

For my tests, I use an EVK board, a Basler connected to CSI1 with ISP0, and I would like to have the OV5640 connected to CSI2, with ISP1.
I have a 1st OV5640 porting on ISP, meaning I can use it alone on CSI1 with ISP0, but I didn't find any way to support 2 devices each one connected on a different ISP module.

BR

[joanxie]

why don't you refer to the dual basler dtb, dual ov2775 basler dtb or basler+ov2775 dtb?

"https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-d...

"https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-d...

"https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-b...

if you just test dual ISP, you can use dual basler , dual ov2775 or balser+ov2775, current bsp already support them

[Charled]

Hi,

I am already based on those dtb files. My dtb is in my original post.

Also, both devices are correctly seen by the Kernel, the corresponding /dev are correctly mounted, and the links are also created. From hw point of view, or at Linux level, it seems everything is fine, but I can just use a single camera, the other one is not seen from Android.

Besides, I don't know if this is an issue, but isp_media_server only opens /dev/video0, never /dev/video1... and I am not even sure the Sensor1_Entry.cfg is then really parsed?

[joanxie]

you just change the dtb file? did you built the isp-imx package for creating camera library? also need to change Sensor1_Entry.cfg or Sensor0_Entry.cfg, you can simulate ov2775 sensor driver in vvcam, but I don't know if all of source code is open to you on android, because some of them are need provided by IP owner

[Charled]

That is effectively an issue. Most of the code is provided pre-compiled, and I have only got an isp-imx 4.2.2.6.0, which I understand is quite old... I have been told to not use any of the generated elements except my own camera library. Do you know which is the latest version of the isp-imx package? Could I simply compile and use it with my Android BSP 11.0.0_1.2.0?

I have done the following:

• Port OV5640 in ISP mode (under vvcam dirctory)
• Modifying the dtb file to declare both camera on ISP.
• Updating both SensorX_Entry.cfg to declare the corresponding library, and the sensor calibration files.
• Adapt the file camera_config_imx8mp.json to declare 2 camera names "viv_v4l2_device" (to ensure they will be mapped on ISP)
• Adapt the makefile to copy all the needed files (calibration files, driver and libraries, config files) on the target

Do you know how isp_media_server can use both devices since it only opens a single /dev/video0? Also, the example I have for its json file in parameter seems to only declare a single link: sensor0 -> dewrap -> v4l2. Do I need to add something for my sensor1? How to do that?

BR

[ederibaucourt]

Hello @Charled , I would like to use the OV5640 with the ISP. Would it be possible for you to share all the files you mention regarding porting this camera for the ISP? Even if they are not fully working, I'd be very interested.

Best Regards,

Enguerrand de Ribaucourt from Savoir-faire Linux

[khang_letruong]

Hello Enguerrand,

This could be something your are looking for : https://github.com/nxp-imx-support/meta-imx8mp-isp-ov5647

Otherwise, you could pm me for the ov5640.

Best Regards/Cordialement,

Khang Le Truong

[khang_letruong]

Hi @Charled ,

From my experience, together with lower-level (i.e. vvcam kernel module + dts) adaptation, you will also need to adapt one of existing example (such as OV2775) in isp-imx-<VERSION>.bin to OV5640. VERSION = 4.2.2.6.0 in your case, so the isp-imx-4.2.2.6.0.bin can be downloaded from following link : https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.6.0.bin

And the OV2775 resources are as below :

units/isi/drv/OV2775/include_priv/OV2775_priv.h

units/isi/drv/OV2775/source/OV2775.c
units/isi/drv/OV2775/source/OV2775_mipi4lane_1080p_3dol_init.txt
units/isi/drv/OV2775/source/OV2775_mipi4lane_1080p_init.txt
units/isi/drv/OV2775/source/OV2775_mipi4lane_1080p_2dol_init.txt

units/isi/drv/OV2775/calib/OV2775/OV2775_pentaxcombined_01.xml
units/isi/drv/OV2775/calib/OV2775/OV2775_8M_02_720p.xml
units/isi/drv/OV2775/calib/OV2775/OV2775.xml
units/isi/drv/OV2775/calib/OV2775/OV2775_8M_02_1080p.xml
units/isi/drv/OV2775/calib/OV2775/OV2775_fisheye.xml
units/isi/drv/OV2775/calib/OV2775/OV2775_pentax_04.xml

You can rename above files to OV5640 or duplicate and rename them to OV5640.

Above resources will allow to generate .drv file which is in turn used by the isp_media_server as an intermediate layer to interface with vvcam kernel module via ioctl() calls.

Also in above self-extracted binary isp-imx-4.2.2.6.0.bin, there will be a need to adapt the scripts imx/start_isp.sh and imx/run.sh to configure the input files of isp_media_server (such as Sensor0_Entry.cfg, Sensor1_Entry.cfg, etc. ) so that it can recognize and load your ISP based (or vvcam) ov5640 driver.

And in imx/run.sh of isp-imx-4.2.2.6.0, only single camera is handled. But you could inspire the following snippet to handle mix of basler camera and ov2775 (or ov5640 in your case ) :

 basler_1080p60_ov2775_1080p30 )
     MODULES=("basler-camera-driver-vvcam" "${MODULES[@]}")
     JSON_FILE="case/config/daA3840_30mc_1080P.json"
     CAM_NAME="basler-vvcam"
     DRV_FILE="DAA3840_30MC_1080P.drv"
     #XML_FILE="DAA3840_30MC_1080P.xml"
     MODE_FILE="DAA3840_MODES.txt"
     MODE="1"
     write_sensor_cfg_file "Sensor0_Entry.cfg" $CAM_NAME $DRV_FILE $MODE_FILE $MODE
     
     MODULES=("ov2775" "${MODULES[@]}")
     JSON_FILE="case/config/2006_sensor_dwe.json"
     CAM_NAME="ov2775"
     DRV_FILE="ov2775.drv"
     #XML_FILE="OV2775.xml"
     MODE_FILE="OV2775_MODES.txt"
     MODE="0"
     write_sensor_cfg_file "Sensor1_Entry.cfg" $CAM_NAME $DRV_FILE $MODE_FILE $MODE

Note that Sensor0_Entry.cfg or Sensor1_Entry.cfg will be re-generated in runtime by the above imx/run.sh.

Beside, you could migrate to more recent BSPs (which include newer isp-imx of course) that provide better examples of dual ISP sensors of same type (not sure if different/mix types are also supported).

Regards,

K.

[Charled]

Thanks @khang_letruong,

I have done this, already. The difference is that the scripts your mention (run.sh, start_isp.sh) are dedicated to Linux, and not used on Android.

I already adapted SensorX_Entry.cfg files.

BR

[khang_letruong]

Hi @Charled ,

I haven't worked with Android for NXP's iMX platforms yet. But I think similar to their Linux (Yocto), there should be a way to re-compile the source code from scratch for custom use-case like yours. Please have a look at this : https://www.nxp.com/docs/en/user-guide/ANDROID_USERS_GUIDE.pdf

Also, isp-imx-4.2.2.6.0 seems to be very initial version. You can try newer Android versions here : https://www.nxp.com/design/software/embedded-software/i-mx-software/android-os-for-i-mx-applications...

Best regards,

Khang