topic Re: IMX8M-Plus Camera Integration and using ISP in i.MX Processors

IMX8M-Plus Camera Integration and using ISP

jgsandom — Mon, 12 Apr 2021 17:40:08 GMT

Hi there,

For reference, version and BSP being used is imx-5.4.70-2.3.0. https://source.codeaurora.org/external/imx/imx-manifest/tree/imx-5.4.70-2.3.0.xml?h=imx-linux-zeus

We're attempting to integrate a camera sensor with the IMX8M-Plus and utilise the on-board ISP. However, we're having some difficulty with this.

It seems as though to get the ISP working a few elements are required;

Enable ISP in DTS
Enable the ISP VVCAM kernel module (kernel-module-isp-vvcam)
Enable the ISP software (isp-imx) - providing imx8-isp.service
Specific .drv and .xml files for the sensor being used

Steps 1-3 seem ok. However, step 4 is a bit more unknown. NXP seem to provide default .drv and .xml files for specific sensors (looking at the Yocto layers for meta-imx), the support seems to be for OV2775 and Basler Camera. Without the correct .drv and .xml files the imx8-isp.service fails to start. Is there any information on how these files can be generated when integrating new camera sensors? Additionally, is there source code available for isp_media_server?

Is there any documentation for porting a new camera and using the ISP on the IMX8M-Plus?

Any support would be appreciated!

Apologies for the label, there doesn't seem to be a label for the IMX8M-Plus yet.

IMX8M-Plus Camera Integration and using ISP

jgsandom — Wed, 14 Apr 2021 10:37:35 GMT

Is anyone able to help? @igorpadykov @joanxie

Re: IMX8M-Plus Camera Integration and using ISP

igorpadykov — Thu, 15 Apr 2021 00:40:27 GMT

Hi Joe

one can start with basler software package

https://www.baslerweb.com/en/sales-support/downloads/software-downloads/basler-camera-enablement-package-fuer-nxps-i-mx-8m-plus-prozessor-ga/

https://www.nxp.com/design/development-boards/i-mx-evaluation-and-development-boards/4k-dart-bcon-for-mipi-camera-module-for-i-mx-8m-plus-applications-processor:MIPI-CAMERA

Best regards
igor

Re: IMX8M-Plus Camera Integration and using ISP

joanxie — Thu, 15 Apr 2021 03:25:39 GMT

I have already mailed to you, pls refer to that

Re: IMX8M-Plus Camera Integration and using ISP

khang_letruong — Thu, 15 Apr 2021 08:38:23 GMT

Dear @joanxie ,

Could you share some info via private message as well since we were already in the Beta Program scope for the iMX8MP but we had not really received enough support for the ISP.

Thanks in advance and best regards,
Khang.

Re: IMX8M-Plus Camera Integration and using ISP

jgsandom — Fri, 23 Apr 2021 14:10:48 GMT

I've emailed @joanxie separately, but if anyone else knows the answer to this I would appreciate it. I believe the ISP should be discussed openly now as the documentation has been officially released.

I've set up the basic structure for my sensor but it seems to get stuck locating the .drv file. The "loading" comment suggests that szDriverName is empty. See snippet below;

INFO : [ISP_MEDIA_SERVER] ************************************************************ INFO : [ISP_MEDIA_SERVER] VIV ISP Media Control Framework V4.2.1 INFO : [ISP_MEDIA_SERVER] ************************************************************ INFO : [MediaPipeline] load: Loading szFileName: case/config/2005_mc_sensor.json INFO : [NativeSensor] Open Device: /dev/video0 (fd=6) INFO : [NativeSensor] Driver: viv_v4l2_device INFO : [NativeSensor] found viv video dev /dev/video0 INFO : [MediaPipeline] add component NativeSensor INFO : [MediaPipeline] add component V4l2Control DEBUG : [MediaPipeline] stream caps : 0 format: 2 w:1280 h:720 DEBUG : [MediaPipeline] stream caps : 0 format: 2 w:1920 h:1080 DEBUG : [MediaPipeline] stream caps : 0 format: 0 w:1280 h:720 DEBUG : [MediaPipeline] stream caps : 0 format: 0 w:1920 h:1080 DEBUG : [MediaPipeline] stream caps : 0 format: 1 w:1280 h:720 DEBUG : [MediaPipeline] stream caps : 0 format: 1 w:1920 h:1080 INFO : [V4l2Event] Open Device: /dev/video0 (fd=5) INFO : [V4l2Event] Driver: viv_v4l2_device INFO : [V4l2Event] found viv video dev /dev/video0 INFO : [MediaPipeline] MEDIA_EVENT_PASS_JSON INFO : [NativeSensor] jsonRequest 98331 loading []...

Looking at the source code (`units/cam_device/source/camera/cam_sensor.cpp`) - I can't see the implementation for HalGetSensorDrvName as the implementation is proprietary. Using GDB I can see the implementation here: `/backup/build/users/nxa18621/vsi-isp/verisilicon_sw_isp/units/hal/source/hal_altera_pci.c:1720`. Does anyone know how this works? the .drv file for my sensor is present. I tried this with OV2775 and had the same issue as well.

Why does HalGetSensorDrvName return an empty string? Also noticed this with HalGetSensorCalibXmlName. Is there a specific directory the files need to be in?

Re: IMX8M-Plus Camera Integration and using ISP

jgsandom — Mon, 26 Apr 2021 08:09:39 GMT

Anyone have any ideas? @joanxie @igorpadykov

Re: IMX8M-Plus Camera Integration and using ISP

joanxie — Mon, 26 Apr 2021 10:42:16 GMT

the document has been released, you can download it

"https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-8-processors/i-mx-8m-plus-arm-cortex-a53-machine-learning-vision-multimedia-and-industrial-iot:IMX8MPLUS?tab=Documentation_Tab"

Re: IMX8M-Plus Camera Integration and using ISP

joanxie — Mon, 26 Apr 2021 10:42:38 GMT

I have mailed to you already

Re: IMX8M-Plus Camera Integration and using ISP

khang_letruong — Wed, 19 May 2021 03:23:43 GMT

Hi,

My personal opinion is that in the Basler CEP, aside from the kernel patches and im8-isp service's scripts that are provided as source code, the rest (including the ISP based camera driver for both VVCAM and ISI layers) is pre-built. It is also true in the isp-imx and isp-vvcam frameworks provided by NXP :

- In ISI layer, basler camera driver is prebuilt :

- In isp-vvcam, there's only camera-proxy-driver to 'speak' with Basler camera modules :

So I think the camera is only useful for purpose of demonstrating the iMX8MP's ISP capacity as well as for ones expecting to use the camera as-is while most of us here have the need of customization I guess. But the documentation is quite in high level to be mapped into the current source code.

Regards,

Khang

Re: IMX8M-Plus Camera Integration and using ISP

jgsandom — Wed, 26 May 2021 20:54:29 GMT

For anyone stuck on this, I managed to get past these issues and get the ISP working with our image sensor using ISP version 4.2.2.11 from BSP 5.4.70-2.3.2: https://source.codeaurora.org/external/imx/meta-imx/tree/meta-bsp/recipes-bsp/isp-imx/isp-imx_4.2.2.11.0.bb?h=zeus-5.4.70-2.3.2

With this version, the .drv file is loaded correctly.

Re: IMX8M-Plus Camera Integration and using ISP

khang_letruong — Fri, 23 Jul 2021 15:23:50 GMT

Dear @jgsandom ,

It seems that I have similar issue as you'd had before. However, I'm working with the isp-imx 4.2.2.11.0. I described my issue here : https://community.nxp.com/t5/i-MX-Processors/IMX8MP-ISP-isp-media-server-crashes-at-camdev-SensorOps/m-p/1312376

Also, I could not re-build with full option but only partial option. I saw in another discussion that one could build with full but it was not true. In that link, someone used Full instead full, but this is not taken into account by the build-all-isp.sh script.

Could you help to debug the issue, please ?

Thanks so much in advance,
Khangg

Re: IMX8M-Plus Camera Integration and using ISP

meetgandhi-eic — Fri, 16 Feb 2024 11:41:21 GMT

Hi,

I'm trying to bring-up basler-daa3840-mc on iMx8 MP solidrun SoM with Ubuntu 20.04 rootfs. I'm using kernel version 5.10.72 which is built from source provided by solidrun with basler camera support. (ref). I'm using the isp-vvcam driver with the same version lf-5.10.y_2.2.0. I'm using isp-imx_4.2.2.16.0 and the patches/overlays provided by basler to support the basler camera.

The camera driver gets probed successfully but when I try to run the imx_media_server application, It gives segmentation fault without printing anything on the console.

Here is the full dump of the start_isp.sh script:

analog@aditof:~/isp-imx-4.2.2.16.0/mediacontrol/daemon$ sudo /opt/imx8-isp/bin/start_isp.sh
Starting isp_media_server
DAA3840_30MC_266 on ISP 1
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Apply configuration isp0: ...
Apply configuration isp1: DAA3840_30MC_266 ...
Loading module basler-camera-driver-vvcam ...
basler_camera_driver_vvcam
basler_camera_driver_vvcam 24576 0
basler-camera-driver-vvcam already loaded.
Loading module imx8-media-dev ...
imx8_media_dev
Loaded /usr/lib/modules/5.10.72-adi-gc8f61459963d/kernel/drivers/staging/media/imx/imx8-media-dev.ko
Loading module vvcam-video ...
vvcam_video
vvcam_video 45056 0
vvcam-video already loaded.
Loading module vvcam-dwe ...
vvcam_dwe
vvcam_dwe 28672 2
vvcam-dwe already loaded.
Loading module vvcam-isp ...
vvcam_isp
vvcam_isp 69632 2
vvcam-isp already loaded.
Starting isp_media_server with configuration CAMERA1
/opt/imx8-isp/bin/run.sh: line 178: 1500 Segmentation fault ./isp_media_server $STARTMODE

Here is the dump of dmesg when after probing the driver before running the script:

[ +0.035842] basler_camera_driver_vvcam: loading out-of-tree module taints kernel.
[ +0.069022] ABRM FamilyName not supported
[ +0.031117] ABRM: Manufacturer: Basler, Model: daA3840-30mc, Device: 108220-05, Serial: basler:40097239
[ +0.056186] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[ +0.001711] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[ +0.000203] mx8-img-md: Registered sensor subdevice: basler-camera-vvcam 3-0036 (1)
[ +0.000012] mx8-img-md: Registered sensor subdevice: adsd3500 1-0038 (2)
[ +0.000009] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]
[ +0.000005] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[ +0.000006] mx8-img-md: created link [adsd3500 1-0038] => [mxc-mipi-csi2.0]
[ +0.000010] mx8-img-md: created link [basler-camera-vvcam 3-0036] => [mxc-mipi-csi2.1]
[ +0.000007] mxc-md 32c00000.bus:camera: mxc_md_create_links
[ +0.072911] enter viv_video_init_module
[ +0.055660] enter viv_dwe_init_module
[ +0.000458] enter dwe_hw_probe
[ +0.000659] vvcam dewarp driver probed
[ +0.062509] enter viv_isp_init_module
[ +0.000445] enter isp_hw_probe
[ +0.000329] vvcam isp driver registered
[ +0.001521] enter isp_hw_probe
[ +0.002429] vvcam isp driver registered
[ +0.009074] enter isp_mi_stop
[ +0.007432] enter isp_mi_stop

Here is the dmesg dump after running the script:

[Feb16 11:04] : Unregistered all entities
[ +0.074758] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[ +0.001282] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[ +0.000098] mx8-img-md: Registered sensor subdevice: basler-camera-vvcam 3-0036 (1)
[ +0.000006] mx8-img-md: Registered sensor subdevice: adsd3500 1-0038 (2)
[ +0.000007] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]
[ +0.000005] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[ +0.000005] mx8-img-md: created link [adsd3500 1-0038] => [mxc-mipi-csi2.0]
[ +0.000004] mx8-img-md: created link [basler-camera-vvcam 3-0036] => [mxc-mipi-csi2.1]
[ +0.000005] mxc-md 32c00000.bus:camera: mxc_md_create_links

Can anyone help to fix the issue?

Re: IMX8M-Plus Camera Integration and using ISP

khang_letruong — Fri, 16 Feb 2024 23:55:07 GMT

Hi @meetgandhi-eic,

Did it work with Yocto based rootfs? Were the .xml file and .json present? Were you using single camera and was it connected to the first MIPI-CSI2 port ?

Regards,

Khang

Re: IMX8M-Plus Camera Integration and using ISP

meetgandhi-eic — Mon, 19 Feb 2024 06:45:04 GMT

Hi @khang_letruong

Did it work with Yocto based rootfs? Were you using single camera and was it connected to the first MIPI-CSI2 port ?

Yes. I tried the yocto based rootfs provided by solidrun from below link.

https://solidrun.atlassian.net/wiki/spaces/developer/pages/315588609/HummingBoard+Pulse+i.MX8M+Plus+-+Basler+Camera+Quick+Start+Guide

Yes. The camera was attached on the same CSI-2 port and I was able to capture the images.

Were the .xml file and .json present?

I'm not sure which xml/json file you're mentioning here. May be I'm missing something here.

Re: IMX8M-Plus Camera Integration and using ISP

khang_letruong — Mon, 19 Feb 2024 07:23:49 GMT

Yes. The camera was attached on the same CSI-2 port and I was able to capture the images.

> Could you compare the log especially in terms of the probing of the camera devices ?

I'm not sure which xml/json file you're mentioning here. May be I'm missing something here.

> The .xml file is the initial ISP configuration for the Basler camera. For the pure Yocto based firmware, it should be found under /opt/imx8-isp/bin/DAA3840_30MC_xxx.xml. For the Ubuntu based rootfs, the path could be different.

The .json file is the dewarping configuration. For the pure Yocto based firmware, it should be found under /opt/imx8-isp/bin/dewarp/. For the Ubuntu based rootfs, the path could be different and the json might not be found by the relevant scripts of Basler as they seem to provide the resources for the pure Yocto based bitbaking system.

Regards,
Khang

Re: IMX8M-Plus Camera Integration and using ISP

meetgandhi-eic — Mon, 19 Feb 2024 13:41:26 GMT

Hi @khang_letruong

> Could you compare the log especially in terms of the probing of the camera devices ?

- I compared the dmesg logs when camera driver is probed. They are same.

> The .json file is the dewarping configuration. For the pure Yocto based firmware, it should be found under /opt/imx8-isp/bin/dewarp/. For the Ubuntu based rootfs, the path could be different and the json might not be found by the relevant scripts of Basler as they seem to provide the resources for the pure Yocto based bitbaking system.

- Regarding the json and xml files, yes they're present in the same directory that you have mentioned. They were installed basler packages which can be found here:

isp-imx overlays:

https://github.com/basler/meta-basler-imx8/tree/hardknott-5.10.72-2.2.0/recipes-imx8mp/isp-imx/files/isp-imx-basler

config file packages:

https://artifacts.baslerweb.com/artifactory/embedded-vision-public/packages/

I have compared the config files and they're different. May be because of different versions. Do you think that could make any impact?

Re: IMX8M-Plus Camera Integration and using ISP

meetgandhi-eic — Wed, 21 Feb 2024 13:32:28 GMT

Hi,

I have compared the dts source for both the images.

The only difference I was able to found was the basler_0 node and basler_ep_0 endpoint is not added as we have a different sensor connected on that port.

Due to this the isp_0 node is also disabled.

Do you think this could make any impact? Is it possible to enable only one isp endpoint and make the camera work?

Re: IMX8M-Plus Camera Integration and using ISP

khang_letruong — Wed, 21 Feb 2024 16:08:39 GMT

Hi @meetgandhi-eic

Due to this the isp_0 node is also disabled.

The following discussion could be relevant : https://community.nxp.com/t5/i-MX-Processors/ISP-1-only-works-when-ISP-0-is-enabled-Why/m-p/1544319