ISP working in i.MX8M Plus

[kunal_003]

Hi NXP Team,

We came across ISP in i.MX8MPlus, which is used in conjunction with camera. We found the device tree node of the same. However, can anyone help on how to use ISP, when to use and flow of ISP?

Thanks in advance !

Regards,

Kunal

[mb1]

Has anyone managed to get imx219 sensor up and running for imx8mp?

I'd be interested in steps/solution.

[khang_letruong]

Hi @mb1 

Not directly with imx219 but other Sony sensor : mine is imx296 and in this thread is imx378 : https://community.nxp.com/t5/i-MX-Processors/Creating-a-MIPI-CSI-Camera-driver-on-i-MX-8MP-EVK/m-p/1...

Regards,

Khang

[mb1]

Thanks for your reply. I guess this will help.

[OGR]

Hi @jgsandom,

Can you provide me a specification from xml that allows me to configure sensor specific calibration located under isp-imx-module/units/isi/drv/<sensor>/calib,

Thanks, Olga

[jgsandom]

Hi @OGR, unfortunately I do not have a specification in how to configure the xml file for a specific sensor. Only NXP and their partners have this information from what I've been told.

Do you have any specific features you want to tune or enable/disable? I have done some reverse engineering on the calibration file so might be able to help you a bit. 

[joanxie]

•ISP driver and Independent sensor driver interface code.

•This is part of Yocto release now.

•It is self extracting package. Follow below steps to download the code.

  wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.6.0.bin

  ./isp-imx-4.2.2.6.0.bin

•Sample Example code.

  ./isp-imx-4.2.2.6.0/appshell/v4l_drm_test/video_test.cpp

  ./isp-imx-4.2.2.6.0/appshell/vvext/vvext.cpp

[malik_cisse]

Hi Joaxie,

How is the isp-imx-4.2.2.6.0.bin bin content actually intendend to be used?

isp_media_server binary is actually already available in opt/imx8-isp/bin

Should I deploy the bin content on the target system and compile?

Thanks, Malik

[malik_cisse]

Hi Joanxie,

I can add the ISP component to the BSP. They all load fine.

However at some point one need to make custom *.drv and *.xml files available for the ISP to work.

There are prebuilt  *.drv and *.xml files for Basler and OV2775 but how can I produce those files myself for a custom sensor. *.drv is not editable. Its a binary file and *.xml seems also to be automatically generated.

You help on this would be much appreciated.

Thanks, Malik

[jgsandom]

Hi @malik_cisse ,

I might be able to help you with this;

First of all, I recommend you use ISP version 4.2.2.11.0; The 4.2.2.6.0 version does not work from my experience.

wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.11.0.bin

Regarding your question about the XML file, I recommend you copy ov2775.xml or ov2775_4k.xml depending on your input resolution. You can then use the vvext application (built as part of the ISP) to make tweaks to the ISP e.g. black level subtraction, RGB gain, white balance, lens correction etc. Based on your desired changes you can update the XML file for your sensor. This takes some time as there is a lack of documentation in this area.

Now regarding the .drv file, have a look at units/isi/drv/ you'll see a folder for OV2775, copy this folder for your sensor and make all of the modifications for your sensor, this takes a fair bit of time but the general steps are outlined in the ISP user guides. By adding your implementation in the drv folder, this will generate the .drv file when you build.

There is apparently a tool for generating the XML file, but I believe this is restricted to NXP partners and requires use of NXP professional services.

Hope this helps.

[malik_cisse]

Hi jgsandom

How is the isp-imx-4.2.2.11.0.bin bin content actually intendend to be used?

isp_media_server binary is actually already available in opt/imx8-isp/bin as part of the yocto BSP deployment.

Should I deploy the bin content on the target system and compile?

Thanks, Malik

[jgsandom]

Hi @malik_cisse ,

That's correct, isp-imx is already available as part of the Yocto BSP. But if you want to make your own modifications to it for your sensor, then you have to use your own isp-imx. If you plan to use already supported sensors, such as OV2775 and basler-camera then you can just update the recipe for 4.2.2.11 or use the basler camera enablement package (provides recipes).

You can build isp-imx separately outside of Yocto for development and testing. Build steps are;

wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.11.0.bin

chmod +x isp-imx-4.2.2.11.0.bin

./build-all-isp.sh Debug Full

scp -r build_output_debug_Full/opt/imx8-isp target@target-ip:/opt

scp -r build_output_debug_Full/usr/lib/* target@target-ip:/usr/lib/

Hope this helps!

[khang_letruong]

Hi @jgsandom ,

Don't you have following issue of missing directories (they are actually in inner place : units/aaa/proprietories/include/ ) when building with 2nd option = full :

Using full source compile
-- =============================================================================
-- Project description: ISP driver project
-- project bin:/home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/build/obj/units
-- project obj dir:/home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/build/obj/units/obj
-- project src:/home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units
-- Cmake root dir:/home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/../units
LIB_ROOT is at: /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/build/generated
INSTALL_ROOT is at: /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/build/dist
system: UNIX
-- Panasonic face recognition disabled
CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:8 (add_subdirectory):
  add_subdirectory given source "aec" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:9 (add_subdirectory):
  add_subdirectory given source "awb" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:10 (add_subdirectory):
  add_subdirectory given source "af" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:11 (add_subdirectory):
  add_subdirectory given source "adpf" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:12 (add_subdirectory):
  add_subdirectory given source "adpcc" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:13 (add_subdirectory):
  add_subdirectory given source "a2dnr" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:14 (add_subdirectory):
  add_subdirectory given source "a3dnr" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:15 (add_subdirectory):
  add_subdirectory given source "awdr3" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:16 (add_subdirectory):
  add_subdirectory given source "ahdr" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:17 (add_subdirectory):
  add_subdirectory given source "aee" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:18 (add_subdirectory):
  add_subdirectory given source "aflt" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:19 (add_subdirectory):
  add_subdirectory given source "avs" which is not an existing directory.

Thanks in advance and best regards,

K.

[malik_cisse]

Hi Khang,

It seems you either have a problem with your meta-toolchain or your .bin did not extract properly.

When using:
./build-all-isp.sh Release Full

I get following correct output: see attachment

[khang_letruong]

Hi again @malik_cisse ,

Regarding your log file :

./build-all-isp.sh Release Full
BUILD_TYPE: release
BUILD: Full
Partial build --------------------->
Release build --------------------->

The Full seems not correct but rather full (that what I used), then you dropped back to the partial build, even-though the log shows BUILD: Full as-is.

Otherwise, I can also have it built either with

./build-all-isp.sh Release Full

or with

./build-all-isp.sh Release

(When second argument is missing, it is Partial implicitly).

I think that the truly "full" option is reserved for VeriSilicon when they need to re-build everything from scratch since they have the private source code. We do only have the prebuilt libraries of 3a (aaa).

Best regards,

Khang

[khang_letruong]

Hi @malik_cisse .

Yes, it is possible since I haven't setup the SDK generated from Yocto. I tried to build it with stand-alone way  (using separated toolchain) thus the sysroot was missing I guess. I'm using Buildroot.

Thanks for your confirmation anyway. I will try with Yocto.

Best regards,

K.

[malik_cisse]

Hi jgsandom,

Sorry to bother you again. Your hints have been very helpfull so far and I hope other people can benefit from it to.

How to modify *.xml file and how to generate *.drv files is clear thanks to your description.

There is also a *.json file to be passed as argument to the isp_media_server application. 

All these files are meant to configure the ISP, right? Do you know how these files are used in the system and which instance is actually using them?

I have hard time matching the file names with the resources described in the " i.MX 8M Plus Camera Sensor Porting User Guide".

In my understanding, isp_media_server is a demond service that starts upon system boot and configures the ISP pipeline elements and matches them with the sensor driver. Is this understanding correct?

Besides the steps described in this thread and a previous thread from you:

https://community.nxp.com/t5/i-MX-Processors/IMX8M-Plus-Camera-Integration-and-using-ISP/m-p/1260702...

Are there other steps you did to make the ISP work in your case?

Thank you, Malik

[jgsandom]

Hi Malik,

No worries at all. Hope this information is helpful to yourself and others!

In terms of the .json file being used with isp_media_server, in the version I'm referring to (4.2.2.11.0), the argument being parsed is not the .json file, it is the $RUN_OPTION;

./isp_media_server $RUN_OPTION

The run option, is either CAMERA0, CAMERA1 or DUAL_CAMERA. Use camera 0 for ISP0, camera 1 for ISP1 or dual camera for 2 cameras.

The .json file is the config for the dewarp engine, examples of these are located in dewarp/dewarp_config. These are fairly generic, and the config is quite basic - I would suggest to just copy the OV2775 dewarp config for either 720/1080/4k (depending on your resolution). You select the dewarp config by specifying in the mode file (this is dynamically written when you run imx/run.sh).

In terms of the daemon and the flow;

imx8-isp.service --> start_isp.sh --> run.sh --> isp_media_server

Essentially, start_isp.sh just does some sanity checks and calls run.sh with a config argument, this will be "your sensor", run.sh loads the kernel module for your sensor + all other necessary modules such as imx8-media-dev, vvcam-isp, vvcam-dwe. Additionally, sets up the mode files and sensor config, providing the paths to your sensor .xml file, .drv file, dewarp config etc.

Once all of this is setup, isp_media_server is run and has all of the necessary components to form the command and control for your sensor.

Some other tips I can give you are, when running isp_media_server you can do export ISP_LOG_LEVEL=3 (I can't remember the exact log level range, think it's 0 to 5) to get some verbose logging of the ISP in action.

You can also see various "TRACERS" in the source code, you can enable the tracer of the image sensor interface tracer i.e. in CREATE_TRACER(ISI_INFO ...) by changing 0 to 1. This gives a lot of good info for seeing exactly what ISI functions are called when the ISP is running.

Hope this helps!

Thanks,

Joe Sandom

[malik_cisse]

Hi Jgsandom,

What you say is 100% clear and I could replicate the steps to a certain extend.

I get one error (see attached image) when running:

./build-all-isp.sh Debug Full

Everything builds fine except isp_media_server. Did you have this issue also?

I could successfully build the release version of the above script however the debug prints are missing.

Are you actually using <STRONG>kernel-module-isp-vvcam version 4.2.2.6.0 </STRONG> together with <STRONG>isp-imx version 4.2.2.11.0 </STRONG>  or did you also update kernel-module-isp-vvcam version from 6  to 11?

You did not mention your actual sensor i2c kernel driver. Did you build your driver as a loadable kernel module similarly to ov2775 in kernel-module-isp-vvcam or is your driver placed in kernel_source/media/driver/i2c together with other regular sensor driver?

Thank you

[jgsandom]

Hi @malik_cisse ,

I didn't get the same error when building in debug, not too sure about the error you attached unfortunately. Possibly due to your toolchain? did you try to do export LDFLAGS= after sourcing your toolchain?

Yes, when I changed isp-imx to revision 4.2.2.11, I also changed kernel-module-isp-vvcam version to 4.2.2.11, this is mandatory or you'll face compatibility issues.

My sensor driver was originally in the kernel source tree, but I moved it to kernel-module-isp-vvcam for the ISP integration because you need to modify your sensor driver e.g. all of the VVSENSORIOC IOCTLS and the sensor mode definition etc. Then I modified the run script to load and unload my sensor module.

Hope this helps!

[malik_cisse]

Hi jgsandom,

Thank you. This is clear.

Actually one does not really need the debug build of isp-imx since release already has all needed features when activating traces and exporting the logs with e.g export ISP_LOG_LEVEL=8.

I am still stuck in making the ISP work though.

You said previously that one should built isp-imx-4.2.2.11.0 outside of Yocto and copy the binaries to the target. Is your yocto still building with isp-imx-4.2.2.6.0 then or did you update that too?

I have the feeling that only building isp-imx-4.2.2.11.0 outside of Yocto (while isp-imx-4.2.2.6.0 is being built in BSP) leaves some older version files in the target.

I get this error when doing your steps:

root@phyboard-pollux-imx8mp-1:bin# ./run.sh -c ov2775_1080p30
realpath: -s: No such file or directory
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "ov2775_1080p30"...
Starting isp_media_server with configuration file CAMERA0
ERROR : [MediaPipeline] NativeSensor open error!
ERROR : [V4l2Event] initialize MediaPipeline error!

I managed to update kernel-module-isp-vvcam version to 4.2.2.11.0 in yocto but I did not succeed trying to incorporate isp-imx-4.2.2.11.0 yet. I keep on getting license and compile issues.

Thank you,