IMX8M-Plus Camera Integration and using ISP

[jgsandom]

Hi there,

For reference, version and BSP being used is imx-5.4.70-2.3.0. https://source.codeaurora.org/external/imx/imx-manifest/tree/imx-5.4.70-2.3.0.xml?h=imx-linux-zeus

We're attempting to integrate a camera sensor with the IMX8M-Plus and utilise the on-board ISP. However, we're having some difficulty with this.

It seems as though to get the ISP working a few elements are required;

1. Enable ISP in DTS
2. Enable the ISP VVCAM kernel module (kernel-module-isp-vvcam)
3. Enable the ISP software (isp-imx) - providing imx8-isp.service
4. Specific .drv and .xml files for the sensor being used

Steps 1-3 seem ok. However, step 4 is a bit more unknown. NXP seem to provide default .drv and .xml files for specific sensors (looking at the Yocto layers for meta-imx), the support seems to be for OV2775 and Basler Camera. Without the correct .drv and .xml files the imx8-isp.service fails to start. Is there any information on how these files can be generated when integrating new camera sensors? Additionally, is there source code available for isp_media_server?

Is there any documentation for porting a new camera and using the ISP on the IMX8M-Plus?

Any support would be appreciated!

Apologies for the label, there doesn't seem to be a label for the IMX8M-Plus yet.

[meetgandhi-eic]

Hi,

I'm trying to bring-up basler-daa3840-mc on iMx8 MP solidrun SoM with Ubuntu 20.04 rootfs. I'm using kernel version 5.10.72 which is built from source provided by solidrun with basler camera support. (ref). I'm using the isp-vvcam driver with the same version lf-5.10.y_2.2.0. I'm using isp-imx_4.2.2.16.0 and the patches/overlays provided by basler to support the basler camera.

The camera driver gets probed successfully but when I try to run the imx_media_server application, It gives segmentation fault without printing anything on the console.

Here is the full dump of the start_isp.sh script:

analog@aditof:~/isp-imx-4.2.2.16.0/mediacontrol/daemon$ sudo /opt/imx8-isp/bin/start_isp.sh
Starting isp_media_server
DAA3840_30MC_266 on ISP 1
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Apply configuration isp0: ...
Apply configuration isp1: DAA3840_30MC_266 ...
Loading module basler-camera-driver-vvcam ...
basler_camera_driver_vvcam
basler_camera_driver_vvcam 24576 0
basler-camera-driver-vvcam already loaded.
Loading module imx8-media-dev ...
imx8_media_dev
Loaded /usr/lib/modules/5.10.72-adi-gc8f61459963d/kernel/drivers/staging/media/imx/imx8-media-dev.ko
Loading module vvcam-video ...
vvcam_video
vvcam_video 45056 0
vvcam-video already loaded.
Loading module vvcam-dwe ...
vvcam_dwe
vvcam_dwe 28672 2
vvcam-dwe already loaded.
Loading module vvcam-isp ...
vvcam_isp
vvcam_isp 69632 2
vvcam-isp already loaded.
Starting isp_media_server with configuration CAMERA1
/opt/imx8-isp/bin/run.sh: line 178: 1500 Segmentation fault ./isp_media_server $STARTMODE

Here is the dump of dmesg when after probing the driver before running the script:

[ +0.035842] basler_camera_driver_vvcam: loading out-of-tree module taints kernel.
[ +0.069022] ABRM FamilyName not supported
[ +0.031117] ABRM: Manufacturer: Basler, Model: daA3840-30mc, Device: 108220-05, Serial: basler:40097239
[ +0.056186] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[ +0.001711] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[ +0.000203] mx8-img-md: Registered sensor subdevice: basler-camera-vvcam 3-0036 (1)
[ +0.000012] mx8-img-md: Registered sensor subdevice: adsd3500 1-0038 (2)
[ +0.000009] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]
[ +0.000005] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[ +0.000006] mx8-img-md: created link [adsd3500 1-0038] => [mxc-mipi-csi2.0]
[ +0.000010] mx8-img-md: created link [basler-camera-vvcam 3-0036] => [mxc-mipi-csi2.1]
[ +0.000007] mxc-md 32c00000.bus:camera: mxc_md_create_links
[ +0.072911] enter viv_video_init_module
[ +0.055660] enter viv_dwe_init_module
[ +0.000458] enter dwe_hw_probe
[ +0.000659] vvcam dewarp driver probed
[ +0.062509] enter viv_isp_init_module
[ +0.000445] enter isp_hw_probe
[ +0.000329] vvcam isp driver registered
[ +0.001521] enter isp_hw_probe
[ +0.002429] vvcam isp driver registered
[ +0.009074] enter isp_mi_stop
[ +0.007432] enter isp_mi_stop

Here is the dmesg dump after running the script:

[Feb16 11:04] : Unregistered all entities
[ +0.074758] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[ +0.001282] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[ +0.000098] mx8-img-md: Registered sensor subdevice: basler-camera-vvcam 3-0036 (1)
[ +0.000006] mx8-img-md: Registered sensor subdevice: adsd3500 1-0038 (2)
[ +0.000007] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]
[ +0.000005] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[ +0.000005] mx8-img-md: created link [adsd3500 1-0038] => [mxc-mipi-csi2.0]
[ +0.000004] mx8-img-md: created link [basler-camera-vvcam 3-0036] => [mxc-mipi-csi2.1]
[ +0.000005] mxc-md 32c00000.bus:camera: mxc_md_create_links

Can anyone help to fix the issue? 

[meetgandhi-eic]

Hi,

I have compared the dts source for both the images.

The only difference I was able to found was the basler_0 node and basler_ep_0 endpoint is not added as we have a different sensor connected on that port.

Due to this the isp_0 node is also disabled.

Do you think this could make any impact? Is it possible to enable only one isp endpoint and make the camera work?

[khang_letruong]

Hi @meetgandhi-eic 

Due to this the isp_0 node is also disabled.

The following discussion could be relevant :  https://community.nxp.com/t5/i-MX-Processors/ISP-1-only-works-when-ISP-0-is-enabled-Why/m-p/1544319

[khang_letruong]

Hi @meetgandhi-eic,

Did it work with Yocto based rootfs? Were the .xml file and .json present? Were you using single camera and was it connected to the first MIPI-CSI2 port ?

Regards,

Khang

[meetgandhi-eic]

Hi @khang_letruong 

Did it work with Yocto based rootfs? Were you using single camera and was it connected to the first MIPI-CSI2 port ?

Yes. I tried the yocto based rootfs provided by solidrun from below link.

https://solidrun.atlassian.net/wiki/spaces/developer/pages/315588609/HummingBoard+Pulse+i.MX8M+Plus+...

Yes. The camera was attached on the same CSI-2 port and I was able to capture the images.

Were the .xml file and .json present?

I'm not sure which xml/json file you're mentioning here. May be I'm missing something here.

[khang_letruong]

Yes. The camera was attached on the same CSI-2 port and I was able to capture the images.

> Could you compare the log especially in terms of the probing of the camera devices ?

I'm not sure which xml/json file you're mentioning here. May be I'm missing something here.

> The .xml file is the initial ISP configuration for the Basler camera. For the pure Yocto based firmware, it should be found under /opt/imx8-isp/bin/DAA3840_30MC_xxx.xml. For the Ubuntu based rootfs, the path could be different.

The .json file is the dewarping configuration. For the pure Yocto based firmware, it should be found under /opt/imx8-isp/bin/dewarp/. For the Ubuntu based rootfs, the path could be different and the json might not be found by the relevant scripts of Basler as they seem to provide the resources for the pure Yocto based bitbaking system. 

Regards,
Khang

[meetgandhi-eic]

Hi @khang_letruong 

> Could you compare the log especially in terms of the probing of the camera devices ?

- I compared the dmesg logs when camera driver is probed. They are same.

> The .json file is the dewarping configuration. For the pure Yocto based firmware, it should be found under /opt/imx8-isp/bin/dewarp/. For the Ubuntu based rootfs, the path could be different and the json might not be found by the relevant scripts of Basler as they seem to provide the resources for the pure Yocto based bitbaking system.

- Regarding the json and xml files, yes they're present in the same directory that you have mentioned. They were installed basler packages which can be found here: 

isp-imx overlays:

https://github.com/basler/meta-basler-imx8/tree/hardknott-5.10.72-2.2.0/recipes-imx8mp/isp-imx/files...

config file packages:

https://artifacts.baslerweb.com/artifactory/embedded-vision-public/packages/

I have compared the config files and they're different. May be because of different versions. Do you think that could make any impact?

[jgsandom]

Is anyone able to help? @igorpadykov @joanxie

[jgsandom]

I've emailed @joanxie separately, but if anyone else knows the answer to this I would appreciate it. I believe the ISP should be discussed openly now as the documentation has been officially released.

I've set up the basic structure for my sensor but it seems to get stuck locating the .drv file. The "loading" comment suggests that szDriverName is empty. See snippet below;

INFO   : [ISP_MEDIA_SERVER] ************************************************************
INFO   : [ISP_MEDIA_SERVER]        VIV ISP Media Control Framework V4.2.1
INFO   : [ISP_MEDIA_SERVER] ************************************************************

INFO   : [MediaPipeline] load: Loading szFileName: case/config/2005_mc_sensor.json
INFO   : [NativeSensor] Open Device: /dev/video0 (fd=6)
INFO   : [NativeSensor]   Driver: viv_v4l2_device
INFO   : [NativeSensor] found viv video dev /dev/video0
INFO   : [MediaPipeline] add component NativeSensor
INFO   : [MediaPipeline] add component V4l2Control
DEBUG  : [MediaPipeline] stream caps : 0 format: 2 w:1280 h:720
DEBUG  : [MediaPipeline] stream caps : 0 format: 2 w:1920 h:1080
DEBUG  : [MediaPipeline] stream caps : 0 format: 0 w:1280 h:720
DEBUG  : [MediaPipeline] stream caps : 0 format: 0 w:1920 h:1080
DEBUG  : [MediaPipeline] stream caps : 0 format: 1 w:1280 h:720
DEBUG  : [MediaPipeline] stream caps : 0 format: 1 w:1920 h:1080
INFO   : [V4l2Event] Open Device: /dev/video0 (fd=5)
INFO   : [V4l2Event]   Driver: viv_v4l2_device
INFO   : [V4l2Event] found viv video dev /dev/video0
INFO   : [MediaPipeline] MEDIA_EVENT_PASS_JSON
INFO   : [NativeSensor] jsonRequest 98331
loading []...

Looking at the source code (`units/cam_device/source/camera/cam_sensor.cpp`) - I can't see the implementation for HalGetSensorDrvName as the implementation is proprietary. Using GDB I can see the implementation here: `/backup/build/users/nxa18621/vsi-isp/verisilicon_sw_isp/units/hal/source/hal_altera_pci.c:1720`. Does anyone know how this works? the .drv file for my sensor is present. I tried this with OV2775 and had the same issue as well.

Why does HalGetSensorDrvName return an empty string? Also noticed this with HalGetSensorCalibXmlName. Is there a specific directory the files need to be in?

[jgsandom]

For anyone stuck on this, I managed to get past these issues and get the ISP working with our image sensor using ISP version 4.2.2.11 from BSP 5.4.70-2.3.2: https://source.codeaurora.org/external/imx/meta-imx/tree/meta-bsp/recipes-bsp/isp-imx/isp-imx_4.2.2....

With this version, the .drv file is loaded correctly.

[khang_letruong]

Dear @jgsandom ,

It seems that I have similar issue as you'd had before. However, I'm working with the isp-imx 4.2.2.11.0. I described my issue here : https://community.nxp.com/t5/i-MX-Processors/IMX8MP-ISP-isp-media-server-crashes-at-camdev-SensorOps...

Also, I could not re-build with full option but only partial option. I saw in another discussion that one could build with full but it was not true. In that link, someone used Full instead full, but this is not taken into account by the build-all-isp.sh script.

Could you help to debug the issue, please ?

Thanks so much in advance,
Khangg

[jgsandom]

Anyone have any ideas? @joanxie @igorpadykov 

[joanxie]

 I have mailed to you already

[joanxie]

I have already mailed to you, pls refer to that

[khang_letruong]

Dear @joanxie ,

Could you share some info via private message as well since we were already in the Beta Program scope for the iMX8MP but we had not really received enough support for the ISP.

Thanks in advance and best regards,
Khang.

[joanxie]

the document has been released, you can download it

"https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-proces..."

[igorpadykov]

Hi Joe

one can start with basler software package

https://www.baslerweb.com/en/sales-support/downloads/software-downloads/basler-camera-enablement-pac...

https://www.nxp.com/design/development-boards/i-mx-evaluation-and-development-boards/4k-dart-bcon-fo...

Best regards
igor

[khang_letruong]

Hi,

My personal opinion is that in the Basler CEP, aside from the kernel patches and im8-isp service's scripts that are provided as source code, the rest (including the ISP based camera driver for both VVCAM and ISI layers) is pre-built. It is also true in the isp-imx and isp-vvcam frameworks provided by NXP :

- In ISI layer, basler camera driver is prebuilt :

- In isp-vvcam, there's only camera-proxy-driver to 'speak' with Basler camera modules :

So I think the camera is only useful for purpose of demonstrating the iMX8MP's ISP capacity as well as for ones expecting to use the camera as-is while most of us here have the need of customization I guess. But the documentation is quite in high level to be mapped into the current source code.

Regards,

Khang