ISP working in i.MX8M Plus

cancel
Showing results for 
Search instead for 
Did you mean: 

ISP working in i.MX8M Plus

2,679 Views
kunal_003
Contributor III

Hi NXP Team,

 

We came across ISP in i.MX8MPlus, which is used in conjunction with camera. We found the device tree node of the same. However, can anyone help on how to use ISP, when to use and flow of ISP?

 

Thanks in advance !

  

Regards,

Kunal

0 Kudos
27 Replies

1,228 Views
OGR
Contributor I

Hi @jgsandom,

Can you provide me a specification from xml that allows me to configure sensor specific calibration located under isp-imx-module/units/isi/drv/<sensor>/calib,

Thanks, Olga

0 Kudos

954 Views
jgsandom
Contributor III

Hi @OGR, unfortunately I do not have a specification in how to configure the xml file for a specific sensor. Only NXP and their partners have this information from what I've been told.

Do you have any specific features you want to tune or enable/disable? I have done some reverse engineering on the calibration file so might be able to help you a bit. 

0 Kudos

2,483 Views
joanxie
NXP TechSupport
NXP TechSupport

•ISP driver and Independent sensor driver interface code.

•This is part of Yocto release now.

•It is self extracting package. Follow below steps to download the code.

  wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.6.0.bin

  ./isp-imx-4.2.2.6.0.bin

•Sample Example code.

  ./isp-imx-4.2.2.6.0/appshell/v4l_drm_test/video_test.cpp

  ./isp-imx-4.2.2.6.0/appshell/vvext/vvext.cpp

1,939 Views
malik_cisse
Contributor II

Hi Joaxie,

How is the isp-imx-4.2.2.6.0.bin bin content actually intendend to be used?

isp_media_server binary is actually already available in opt/imx8-isp/bin

Should I deploy the bin content on the target system and compile?

Thanks, Malik

0 Kudos

1,980 Views
malik_cisse
Contributor II

Hi Joanxie,

I can add the ISP component to the BSP. They all load fine.

However at some point one need to make custom *.drv and *.xml files available for the ISP to work.

There are prebuilt  *.drv and *.xml files for Basler and OV2775 but how can I produce those files myself for a custom sensor. *.drv is not editable. Its a binary file and *.xml seems also to be automatically generated.

You help on this would be much appreciated.

Thanks, Malik

0 Kudos

1,975 Views
jgsandom
Contributor III

Hi @malik_cisse ,

I might be able to help you with this;

First of all, I recommend you use ISP version 4.2.2.11.0; The 4.2.2.6.0 version does not work from my experience.

wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.11.0.bin

Regarding your question about the XML file, I recommend you copy ov2775.xml or ov2775_4k.xml depending on your input resolution. You can then use the vvext application (built as part of the ISP) to make tweaks to the ISP e.g. black level subtraction, RGB gain, white balance, lens correction etc. Based on your desired changes you can update the XML file for your sensor. This takes some time as there is a lack of documentation in this area.

Now regarding the .drv file, have a look at units/isi/drv/ you'll see a folder for OV2775, copy this folder for your sensor and make all of the modifications for your sensor, this takes a fair bit of time but the general steps are outlined in the ISP user guides. By adding your implementation in the drv folder, this will generate the .drv file when you build.

There is apparently a tool for generating the XML file, but I believe this is restricted to NXP partners and requires use of NXP professional services.

Hope this helps.

 

1,954 Views
malik_cisse
Contributor II

Hi jgsandom

How is the isp-imx-4.2.2.11.0.bin bin content actually intendend to be used?

isp_media_server binary is actually already available in opt/imx8-isp/bin as part of the yocto BSP deployment.

Should I deploy the bin content on the target system and compile?

Thanks, Malik

0 Kudos

1,948 Views
jgsandom
Contributor III

Hi @malik_cisse ,

That's correct, isp-imx is already available as part of the Yocto BSP. But if you want to make your own modifications to it for your sensor, then you have to use your own isp-imx. If you plan to use already supported sensors, such as OV2775 and basler-camera then you can just update the recipe for 4.2.2.11 or use the basler camera enablement package (provides recipes).

You can build isp-imx separately outside of Yocto for development and testing. Build steps are;

wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.11.0.bin

chmod +x isp-imx-4.2.2.11.0.bin

./build-all-isp.sh Debug Full

scp -r build_output_debug_Full/opt/imx8-isp target@target-ip:/opt

scp -r build_output_debug_Full/usr/lib/* target@target-ip:/usr/lib/

Hope this helps!

 

0 Kudos

1,545 Views
khang_letruong
Senior Contributor I

Hi @jgsandom ,

Don't you have following issue of missing directories (they are actually in inner place : units/aaa/proprietories/include/ ) when building with 2nd option = full :

Using full source compile
-- =============================================================================
-- Project description: ISP driver project
-- project bin:/home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/build/obj/units
-- project obj dir:/home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/build/obj/units/obj
-- project src:/home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units
-- Cmake root dir:/home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/../units
LIB_ROOT is at: /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/build/generated
INSTALL_ROOT is at: /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/appshell/build/dist
system: UNIX
-- Panasonic face recognition disabled
CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:8 (add_subdirectory):
  add_subdirectory given source "aec" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:9 (add_subdirectory):
  add_subdirectory given source "awb" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:10 (add_subdirectory):
  add_subdirectory given source "af" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:11 (add_subdirectory):
  add_subdirectory given source "adpf" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:12 (add_subdirectory):
  add_subdirectory given source "adpcc" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:13 (add_subdirectory):
  add_subdirectory given source "a2dnr" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:14 (add_subdirectory):
  add_subdirectory given source "a3dnr" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:15 (add_subdirectory):
  add_subdirectory given source "awdr3" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:16 (add_subdirectory):
  add_subdirectory given source "ahdr" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:17 (add_subdirectory):
  add_subdirectory given source "aee" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:18 (add_subdirectory):
  add_subdirectory given source "aflt" which is not an existing directory.


CMake Error at /home/khang/Workspace/Dev/Dynimlabs/SW/isp-imx/units/aaa/CMakeLists.txt:19 (add_subdirectory):
  add_subdirectory given source "avs" which is not an existing directory.

 

Thanks in advance and best regards,

K.

0 Kudos

1,509 Views
malik_cisse
Contributor II

Hi Khang,

It seems you either have a problem with your meta-toolchain or your .bin did not extract properly.

When using:
./build-all-isp.sh Release Full

 

I get following correct output: see attachment

1,494 Views
khang_letruong
Senior Contributor I

Hi again @malik_cisse ,

Regarding your log file :

 

./build-all-isp.sh Release Full
BUILD_TYPE: release
BUILD: Full
Partial build --------------------->
Release build --------------------->

 

The Full seems not correct but rather full (that what I used), then you dropped back to the partial build, even-though the log shows BUILD: Full as-is.

Otherwise, I can also have it built either with

 

./build-all-isp.sh Release Full

 

or with

 

./build-all-isp.sh Release

 

(When second argument is missing, it is Partial implicitly).

I think that the truly "full" option is reserved for VeriSilicon when they need to re-build everything from scratch since they have the private source code. We do only have the prebuilt libraries of 3a (aaa).

Best regards,

Khang

0 Kudos

1,507 Views
khang_letruong
Senior Contributor I

Hi @malik_cisse .

Yes, it is possible since I haven't setup the SDK generated from Yocto. I tried to build it with stand-alone way  (using separated toolchain) thus the sysroot was missing I guess. I'm using Buildroot.

Thanks for your confirmation anyway. I will try with Yocto.

 

Best regards,

K.

0 Kudos

1,847 Views
malik_cisse
Contributor II

Hi jgsandom,

Sorry to bother you again. Your hints have been very helpfull so far and I hope other people can benefit from it to.

How to modify *.xml file and how to generate *.drv files is clear thanks to your description.

There is also a *.json file to be passed as argument to the isp_media_server application. 

All these files are meant to configure the ISP, right? Do you know how these files are used in the system and which instance is actually using them?

I have hard time matching the file names with the resources described in the " i.MX 8M Plus Camera Sensor Porting User Guide".

In my understanding, isp_media_server is a demond service that starts upon system boot and configures the ISP pipeline elements and matches them with the sensor driver. Is this understanding correct?

Besides the steps described in this thread and a previous thread from you:

https://community.nxp.com/t5/i-MX-Processors/IMX8M-Plus-Camera-Integration-and-using-ISP/m-p/1260702...

Are there other steps you did to make the ISP work in your case?

Thank you, Malik

 

0 Kudos

1,841 Views
jgsandom
Contributor III

Hi Malik,

No worries at all. Hope this information is helpful to yourself and others!

In terms of the .json file being used with isp_media_server, in the version I'm referring to (4.2.2.11.0), the argument being parsed is not the .json file, it is the $RUN_OPTION;

./isp_media_server $RUN_OPTION

The run option, is either CAMERA0, CAMERA1 or DUAL_CAMERA. Use camera 0 for ISP0, camera 1 for ISP1 or dual camera for 2 cameras.

The .json file is the config for the dewarp engine, examples of these are located in dewarp/dewarp_config. These are fairly generic, and the config is quite basic - I would suggest to just copy the OV2775 dewarp config for either 720/1080/4k (depending on your resolution). You select the dewarp config by specifying in the mode file (this is dynamically written when you run imx/run.sh).

In terms of the daemon and the flow;

imx8-isp.service --> start_isp.sh --> run.sh --> isp_media_server

Essentially, start_isp.sh just does some sanity checks and calls run.sh with a config argument, this will be "your sensor", run.sh loads the kernel module for your sensor + all other necessary modules such as imx8-media-dev, vvcam-isp, vvcam-dwe. Additionally, sets up the mode files and sensor config, providing the paths to your sensor .xml file, .drv file, dewarp config etc.

Once all of this is setup, isp_media_server is run and has all of the necessary components to form the command and control for your sensor.

Some other tips I can give you are, when running isp_media_server you can do export ISP_LOG_LEVEL=3 (I can't remember the exact log level range, think it's 0 to 5) to get some verbose logging of the ISP in action.

You can also see various "TRACERS" in the source code, you can enable the tracer of the image sensor interface tracer i.e. in CREATE_TRACER(ISI_INFO ...) by changing 0 to 1. This gives a lot of good info for seeing exactly what ISI functions are called when the ISP is running.

Hope this helps!

Thanks,

Joe Sandom

1,817 Views
malik_cisse
Contributor II

Hi Jgsandom,

What you say is 100% clear and I could replicate the steps to a certain extend.

I get one error (see attached image) when running:

./build-all-isp.sh Debug Full

Everything builds fine except isp_media_server. Did you have this issue also?

I could successfully build the release version of the above script however the debug prints are missing.

Are you actually using <STRONG>kernel-module-isp-vvcam version 4.2.2.6.0 </STRONG> together with <STRONG>isp-imx version 4.2.2.11.0 </STRONG>  or did you also update kernel-module-isp-vvcam version from 6  to 11?

You did not mention your actual sensor i2c kernel driver. Did you build your driver as a loadable kernel module similarly to ov2775 in kernel-module-isp-vvcam or is your driver placed in kernel_source/media/driver/i2c together with other regular sensor driver?

Thank you

0 Kudos

1,790 Views
jgsandom
Contributor III

Hi @malik_cisse ,

I didn't get the same error when building in debug, not too sure about the error you attached unfortunately. Possibly due to your toolchain? did you try to do export LDFLAGS= after sourcing your toolchain?

Yes, when I changed isp-imx to revision 4.2.2.11, I also changed kernel-module-isp-vvcam version to 4.2.2.11, this is mandatory or you'll face compatibility issues.

My sensor driver was originally in the kernel source tree, but I moved it to kernel-module-isp-vvcam for the ISP integration because you need to modify your sensor driver e.g. all of the VVSENSORIOC IOCTLS and the sensor mode definition etc. Then I modified the run script to load and unload my sensor module.

Hope this helps!

0 Kudos

1,712 Views
malik_cisse
Contributor II

Hi jgsandom,

Thank you. This is clear.

Actually one does not really need the debug build of isp-imx since release already has all needed features when activating traces and exporting the logs with e.g export ISP_LOG_LEVEL=8.

I am still stuck in making the ISP work though.

You said previously that one should built isp-imx-4.2.2.11.0 outside of Yocto and copy the binaries to the target. Is your yocto still building with isp-imx-4.2.2.6.0 then or did you update that too?

I have the feeling that only building isp-imx-4.2.2.11.0 outside of Yocto (while isp-imx-4.2.2.6.0 is being built in BSP) leaves some older version files in the target.

I get this error when doing your steps:

root@phyboard-pollux-imx8mp-1:bin# ./run.sh -c ov2775_1080p30
realpath: -s: No such file or directory
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "ov2775_1080p30"...
Starting isp_media_server with configuration file CAMERA0
ERROR : [MediaPipeline] NativeSensor open error!
ERROR : [V4l2Event] initialize MediaPipeline error!

I managed to update kernel-module-isp-vvcam version to 4.2.2.11.0 in yocto but I did not succeed trying to incorporate isp-imx-4.2.2.11.0 yet. I keep on getting license and compile issues.

Thank you,

0 Kudos

1,704 Views
malik_cisse
Contributor II

Hi jgsandom,

Thank you. This is 100% clear.

Actually one does not really need the debug build of isp-imx since release already has all needed features when activating traces and exporting the logs with export ISP_LOG_LEVEL=8.

I am still stuck in making the ISP work though. I think its due to the fact that I let yocto build with older isp-imx-4.2.2.6.0 which I then partly overwrite with separately cross-compiled isp-imx-4.2.2.11.0 using:

scp -r build_output_release_Full/opt/imx8-isp target@target-ip:/opt

scp -r build_output_release_Full/usr/lib/* target@target-ip:/usr/lib/

I have the feeling that some version 6 related stuff is still to be overwritten besides the two lines above.

my run.sh script cannot properly run (open fails).

I think I need to incorporate isp-imx-4.2.2.11.0 to the yocto too.

I tried putting isp-imx_4.2.2.11.0.bb next to isp-imx_4.2.2.6.0.bb and select version 11 in local.conf like this:

IMAGE_INSTALL_append = " \
kernel-module-isp-vvcam \
isp-imx \
"
PREFERRED_VERSION_isp-imx = "4.2.2.11.0"
PREFERRED_VERSION_kernel-module-isp-vvcam = "4.2.2.11.0"

I also tried to setup separate meta-layers but it did not help either.

I have hard time with this step and keep on getting license mismatch errors.

How did you manage to update vvcam and isp user space code from 6 to 11? I am unfortunatelly not a yocto expert.

Thank you,

0 Kudos

1,887 Views
malik_cisse
Contributor II

Thank you so much jgsandom,

This is exactly the type of info that would help moving on with the ISP.

NXP: Please provide a one page tutorial on how to integrate the ISP with custom Sensors not using Basler at all.

Thank you

0 Kudos

1,871 Views
joanxie
NXP TechSupport
NXP TechSupport