i.MX ProcessorsのトピックRe: ISP working in i.MX8M Plus

ISP working in i.MX8M Plus

kunal_003 — Wed, 17 Mar 2021 04:33:50 GMT

Hi NXP Team,

We came across ISP in i.MX8MPlus, which is used in conjunction with camera. We found the device tree node of the same. However, can anyone help on how to use ISP, when to use and flow of ISP?

Thanks in advance !

Regards,

Kunal

Re: ISP working in i.MX8M Plus

joanxie — Wed, 17 Mar 2021 09:29:31 GMT

•ISP driver and Independent sensor driver interface code.

•This is part of Yocto release now.

•It is self extracting package. Follow below steps to download the code.

wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.6.0.bin

./isp-imx-4.2.2.6.0.bin

•Sample Example code.

./isp-imx-4.2.2.6.0/appshell/v4l_drm_test/video_test.cpp

./isp-imx-4.2.2.6.0/appshell/vvext/vvext.cpp

Re: ISP working in i.MX8M Plus

kunal_003 — Thu, 18 Mar 2021 06:31:28 GMT

Hi @joanxie,

Thank you for your response.

Where would I find the document of the bin/source file you shared? Also, where in yocto this source code will be?

We are planning to use Sony IMX219 camera sensor which is RAW sensor. Can you please help what changes are required to use ISP with some example?

Regards,

Kunal

Re: ISP working in i.MX8M Plus

joanxie — Thu, 18 Mar 2021 06:53:46 GMT

this is internal material, I couldn't share here(public community), but I copy the steps here

•Sample code to port new camera Module.

https://source.codeaurora.org/external/imx/isp-vvcam/

Re: ISP working in i.MX8M Plus

kunal_003 — Thu, 01 Apr 2021 05:55:36 GMT

Hi @joanxie,

I saw the link(isp-vvacm) provided by you, however I still have below query/confusion. Please help me in this case:

From what I understand we need to compile this and install the module by running "insmod.sh". So, it will add userspace support for using ISP. Is this understanding correct?
I saw this camera sensors directory, so for SONY IMX219 we need to add support for using ISP?
Can we use ISP only by enabling in device tree?
Any example pipleline or utility to use streaming via ISP?
I am still with many confusions on what exactly needs to be done in order to use ISP. Any documentation/guidance/explanation will be helpful.

Regards,

Kunal

Re: ISP working in i.MX8M Plus

joanxie — Sat, 03 Apr 2021 05:35:59 GMT

current bsp doesn't support SONY IMX219 directly, you need porting by yourself, current bsp includes ISP source code directly, you can find them under path after you building the bsp

"imx-yocto-bsp/build-xwayland/tmp/work/imx8mpevk-poky-linux/kernel-module-isp-vvcam/4.2.2.6.1-r0/git/vvcam"

you also can find the source code from:

"https://source.codeaurora.org/external/imx/isp-vvcam/tree/vvcam?h=imx_5.4.70_2.3.0"

Re: ISP working in i.MX8M Plus

malik_cisse — Thu, 13 May 2021 08:48:36 GMT

Hi Joanxie,

isp-imx-4.2.2.6.0.bin seems indeed to contain all kernel + user space resources needed for ISP.

I would like to buy an EVK + Camera to demonstrate exactly this. What EVK+Camera should I use. I would like to start my own ISP development from a working system.

The Basler BCON cam (official cam of the NXP imx8mp evk) is not using isp-imx-4.2.2.6.0.bin because Basler uses their own user space software (Pylon).

Thank you

Re: ISP working in i.MX8M Plus

joanxie — Thu, 13 May 2021 09:20:54 GMT

we bought this for imx8mp, you can refer to it

“https://www.digikey.cn/en/products/detail/basler-inc/108225/13244085?s=N4IgTCBcDaICYEMEGYAcAWADAWmZgtgMYgC6AvkA”

Re: ISP working in i.MX8M Plus

malik_cisse — Thu, 27 May 2021 18:55:51 GMT

Hi Joanxie,

I can add the ISP component to the BSP. They all load fine.

However at some point one need to make custom *.drv and *.xml files available for the ISP to work.

There are prebuilt *.drv and *.xml files for Basler and OV2775 but how can I produce those files myself for a custom sensor. *.drv is not editable. Its a binary file and *.xml seems also to be automatically generated.

You help on this would be much appreciated.

Thanks, Malik

Re: ISP working in i.MX8M Plus

jgsandom — Thu, 27 May 2021 22:32:56 GMT

Hi @malik_cisse ,

I might be able to help you with this;

First of all, I recommend you use ISP version 4.2.2.11.0; The 4.2.2.6.0 version does not work from my experience.

wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.11.0.bin

Regarding your question about the XML file, I recommend you copy ov2775.xml or ov2775_4k.xml depending on your input resolution. You can then use the vvext application (built as part of the ISP) to make tweaks to the ISP e.g. black level subtraction, RGB gain, white balance, lens correction etc. Based on your desired changes you can update the XML file for your sensor. This takes some time as there is a lack of documentation in this area.

Now regarding the .drv file, have a look at units/isi/drv/ you'll see a folder for OV2775, copy this folder for your sensor and make all of the modifications for your sensor, this takes a fair bit of time but the general steps are outlined in the ISP user guides. By adding your implementation in the drv folder, this will generate the .drv file when you build.

There is apparently a tool for generating the XML file, but I believe this is restricted to NXP partners and requires use of NXP professional services.

Hope this helps.

Re: ISP working in i.MX8M Plus

malik_cisse — Fri, 28 May 2021 06:47:26 GMT

Thank you so much jgsandom.

I am going to investigate your suggestions. This is highly appreciated since discovering all this without help is extremly tedious.

Re: ISP working in i.MX8M Plus

malik_cisse — Fri, 28 May 2021 13:54:26 GMT

Hi Joaxie,

How is the isp-imx-4.2.2.6.0.bin bin content actually intendend to be used?

isp_media_server binary is actually already available in opt/imx8-isp/bin

Should I deploy the bin content on the target system and compile?

Thanks, Malik

Re: ISP working in i.MX8M Plus

malik_cisse — Fri, 28 May 2021 13:56:08 GMT

Hi jgsandom

How is the isp-imx-4.2.2.11.0.bin bin content actually intendend to be used?

isp_media_server binary is actually already available in opt/imx8-isp/bin as part of the yocto BSP deployment.

Should I deploy the bin content on the target system and compile?

Thanks, Malik

Re: ISP working in i.MX8M Plus

jgsandom — Fri, 28 May 2021 16:08:22 GMT

Hi @malik_cisse ,

That's correct, isp-imx is already available as part of the Yocto BSP. But if you want to make your own modifications to it for your sensor, then you have to use your own isp-imx. If you plan to use already supported sensors, such as OV2775 and basler-camera then you can just update the recipe for 4.2.2.11 or use the basler camera enablement package (provides recipes).

You can build isp-imx separately outside of Yocto for development and testing. Build steps are;

wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.11.0.bin
chmod +x isp-imx-4.2.2.11.0.bin
./build-all-isp.sh Debug Full
scp -r build_output_debug_Full/opt/imx8-isp target@target-ip:/opt
scp -r build_output_debug_Full/usr/lib/* target@target-ip:/usr/lib/

Hope this helps!

Re: ISP working in i.MX8M Plus

malik_cisse — Mon, 31 May 2021 07:03:23 GMT

Thank you so much jgsandom,

This is exactly the type of info that would help moving on with the ISP.

NXP: Please provide a one page tutorial on how to integrate the ISP with custom Sensors not using Basler at all.

Thank you

Re: ISP working in i.MX8M Plus

joanxie — Tue, 01 Jun 2021 02:17:57 GMT

we have released the user guide for sensor porting:

"https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-8-processors/i-mx-8m-plus-arm-cortex-a53-machine-learning-vision-multimedia-and-industrial-iot:IMX8MPLUS?tab=Documentation_Tab"

Re: ISP working in i.MX8M Plus

malik_cisse — Tue, 01 Jun 2021 14:23:58 GMT

Hi jgsandom,

Sorry to bother you again. Your hints have been very helpfull so far and I hope other people can benefit from it to.

How to modify *.xml file and how to generate *.drv files is clear thanks to your description.

There is also a *.json file to be passed as argument to the isp_media_server application.

All these files are meant to configure the ISP, right? Do you know how these files are used in the system and which instance is actually using them?

I have hard time matching the file names with the resources described in the " i.MX 8M Plus Camera Sensor Porting User Guide".

In my understanding, isp_media_server is a demond service that starts upon system boot and configures the ISP pipeline elements and matches them with the sensor driver. Is this understanding correct?

Besides the steps described in this thread and a previous thread from you:

https://community.nxp.com/t5/i-MX-Processors/IMX8M-Plus-Camera-Integration-and-using-ISP/m-p/1260702#M172463

Are there other steps you did to make the ISP work in your case?

Thank you, Malik

Re: ISP working in i.MX8M Plus

jgsandom — Tue, 01 Jun 2021 21:56:39 GMT

Hi Malik,

No worries at all. Hope this information is helpful to yourself and others!

In terms of the .json file being used with isp_media_server, in the version I'm referring to (4.2.2.11.0), the argument being parsed is not the .json file, it is the $RUN_OPTION;

./isp_media_server $RUN_OPTION

The run option, is either CAMERA0, CAMERA1 or DUAL_CAMERA. Use camera 0 for ISP0, camera 1 for ISP1 or dual camera for 2 cameras.

The .json file is the config for the dewarp engine, examples of these are located in dewarp/dewarp_config. These are fairly generic, and the config is quite basic - I would suggest to just copy the OV2775 dewarp config for either 720/1080/4k (depending on your resolution). You select the dewarp config by specifying in the mode file (this is dynamically written when you run imx/run.sh).

In terms of the daemon and the flow;

imx8-isp.service --> start_isp.sh --> run.sh --> isp_media_server

Essentially, start_isp.sh just does some sanity checks and calls run.sh with a config argument, this will be "your sensor", run.sh loads the kernel module for your sensor + all other necessary modules such as imx8-media-dev, vvcam-isp, vvcam-dwe. Additionally, sets up the mode files and sensor config, providing the paths to your sensor .xml file, .drv file, dewarp config etc.

Once all of this is setup, isp_media_server is run and has all of the necessary components to form the command and control for your sensor.

Some other tips I can give you are, when running isp_media_server you can do export ISP_LOG_LEVEL=3 (I can't remember the exact log level range, think it's 0 to 5) to get some verbose logging of the ISP in action.

You can also see various "TRACERS" in the source code, you can enable the tracer of the image sensor interface tracer i.e. in CREATE_TRACER(ISI_INFO ...) by changing 0 to 1. This gives a lot of good info for seeing exactly what ISI functions are called when the ISP is running.

Hope this helps!

Thanks,

Joe Sandom

Re: ISP working in i.MX8M Plus

malik_cisse — Wed, 02 Jun 2021 19:29:50 GMT

Hi Jgsandom,

What you say is 100% clear and I could replicate the steps to a certain extend.

I get one error (see attached image) when running:

./build-all-isp.sh Debug Full

Everything builds fine except isp_media_server. Did you have this issue also?

I could successfully build the release version of the above script however the debug prints are missing.

Are you actually using <STRONG>kernel-module-isp-vvcam version 4.2.2.6.0 </STRONG> together with <STRONG>isp-imx version 4.2.2.11.0 </STRONG> or did you also update kernel-module-isp-vvcam version from 6 to 11?

You did not mention your actual sensor i2c kernel driver. Did you build your driver as a loadable kernel module similarly to ov2775 in kernel-module-isp-vvcam or is your driver placed in kernel_source/media/driver/i2c together with other regular sensor driver?

Thank you

Re: ISP working in i.MX8M Plus

jgsandom — Fri, 04 Jun 2021 21:44:52 GMT

Hi @malik_cisse ,

I didn't get the same error when building in debug, not too sure about the error you attached unfortunately. Possibly due to your toolchain? did you try to do export LDFLAGS= after sourcing your toolchain?

Yes, when I changed isp-imx to revision 4.2.2.11, I also changed kernel-module-isp-vvcam version to 4.2.2.11, this is mandatory or you'll face compatibility issues.

My sensor driver was originally in the kernel source tree, but I moved it to kernel-module-isp-vvcam for the ISP integration because you need to modify your sensor driver e.g. all of the VVSENSORIOC IOCTLS and the sensor mode definition etc. Then I modified the run script to load and unload my sensor module.

Hope this helps!