ISP working in i.MX8M Plus

[kunal_003]

Hi NXP Team,

We came across ISP in i.MX8MPlus, which is used in conjunction with camera. We found the device tree node of the same. However, can anyone help on how to use ISP, when to use and flow of ISP?

Thanks in advance !

Regards,

Kunal

[malik_cisse]

Hi jgsandom,

Thank you. This is 100% clear.

Actually one does not really need the debug build of isp-imx since release already has all needed features when activating traces and exporting the logs with export ISP_LOG_LEVEL=8.

I am still stuck in making the ISP work though. I think its due to the fact that I let yocto build with older isp-imx-4.2.2.6.0 which I then partly overwrite with separately cross-compiled isp-imx-4.2.2.11.0 using:

scp -r build_output_release_Full/opt/imx8-isp target@target-ip:/opt

scp -r build_output_release_Full/usr/lib/* target@target-ip:/usr/lib/

I have the feeling that some version 6 related stuff is still to be overwritten besides the two lines above.

my run.sh script cannot properly run (open fails).

I think I need to incorporate isp-imx-4.2.2.11.0 to the yocto too.

I tried putting isp-imx_4.2.2.11.0.bb next to isp-imx_4.2.2.6.0.bb and select version 11 in local.conf like this:

IMAGE_INSTALL_append = " \
kernel-module-isp-vvcam \
isp-imx \
"
PREFERRED_VERSION_isp-imx = "4.2.2.11.0"
PREFERRED_VERSION_kernel-module-isp-vvcam = "4.2.2.11.0"

I also tried to setup separate meta-layers but it did not help either.

I have hard time with this step and keep on getting license mismatch errors.

How did you manage to update vvcam and isp user space code from 6 to 11? I am unfortunatelly not a yocto expert.

Thank you,

[malik_cisse]

Thank you so much jgsandom,

This is exactly the type of info that would help moving on with the ISP.

NXP: Please provide a one page tutorial on how to integrate the ISP with custom Sensors not using Basler at all.

Thank you

[joanxie]

we have released the user guide for sensor porting:

"https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-proces..."

[malik_cisse]

Thank you so much jgsandom.

I am going to investigate your suggestions. This is highly appreciated since discovering all this without help is extremly tedious.

[malik_cisse]

Hi Joanxie,

isp-imx-4.2.2.6.0.bin seems indeed to contain all kernel + user space resources needed for ISP.

I would like to buy an EVK + Camera to demonstrate exactly this. What EVK+Camera should I use. I would like to start my own ISP development from a working system.

The Basler BCON cam (official cam of the NXP imx8mp evk) is not using isp-imx-4.2.2.6.0.bin because Basler uses their own user space software (Pylon).

Thank you

[joanxie]

we bought this for imx8mp, you can refer to it

“https://www.digikey.cn/en/products/detail/basler-inc/108225/13244085?s=N4IgTCBcDaICYEMEGYAcAWADAWmZg...

[kunal_003]

Hi @joanxie,

Thank you for your response.

Where would I find the document of the bin/source file you shared? Also, where in yocto this source code will be?

We are planning to use Sony IMX219 camera sensor which is RAW sensor. Can you please help what changes are required to use ISP with some example?

Regards,

Kunal

[Amal_Antony3331]

Hi @kunal_003 

Is it possible to know which all are the basic registers that need to be configured for IMX219 sensor?

I have configured register sets which are mentioned in the open source IMX219 driver.(https://github.com/torvalds/linux/blob/master/drivers/media/i2c/imx219.c) .

How can we verify whether all the registers are configured properly. When i tried with i2cdump getting the following output/

root@imx8mpevk:~# i2cdump -f -y 2 0x10 w
     0,8  1,9  2,a  3,b  4,c  5,d  6,e  7,f
00: 0000 0000 0000 0004 0000 0000 0000 0000 
08: 0000 0000 0000 ff03 0000 0000 0000 0000 
10: 0000 0000 0000 0077 0000 0000 0000 0000 
18: 0000 0000 0000 0000 0000 0000 0000 0000 
20: 0000 0000 0000 0000 0000 0000 0000 0000 
28: 0000 0000 0000 0000 0000 0000 0000 0000 
30: 0000 0000 0000 02a8 0000 ec7b 7558 0000 
38: 0000 0000 0000 0000 0000 0000 0000 0000 
40: 0000 0000 0000 00ff 0000 0000 0000 0000 
48: 0000 0000 0000 0000 0000 0000 0000 0000 
50: 0000 0000 0000 0f00 0000 0000 0000 0000 
58: 0000 0000 0000 0000 0000 0000 0000 0000 
60: 0000 0000 0000 0000 0000 0000 0000 0000 
68: 0000 0000 0000 0000 0000 0000 0000 0000 
70: 0000 0000 0000 0000 0000 aa1b bb2d 0000 
78: 0000 0000 0000 0000 0000 0000 0000 0100 
80: 0000 0000 0000 0000 0000 0000 0000 0000 
88: 0000 0000 0000 0000 0000 0000 0000 0300 
90: 0008 0000 0000 0000 0000 0000 0000 0000 
98: 0000 0000 0000 0000 0000 0000 0000 0000 
a0: 0000 0000 0000 0000 0000 0000 0000 0000 
a8: 0000 0000 0000 0000 0000 0000 0000 0000 
b0: 0000 0000 0000 0000 0000 06ee 7b4e 0000 
b8: 0000 0000 0000 0000 0000 0000 0000 0000 
c0: 0000 0000 0000 019f 0000 0000 0000 0000 
c8: 0000 0000 0000 0000 0000 0000 0000 0300 
d0: 0000 0000 0000 0000 0000 0000 0000 0000 
d8: 0000 0000 0000 0000 0000 0000 0000 0000 
e0: 0000 0000 0000 0000 0000 0000 0000 0000 
e8: 0000 0000 0000 0000 0000 0000 0000 0000 
f0: 0000 0000 0000 0001 a38b b769 89a0 0000 
f8: 0000 0000 0000 0000 0000 0000 0000 0000 
root@imx8mpevk:~# 

It is not matching with the values in the driver code. Is there any other ways to verify this.

Can you please help here.

Thanks in advance

Best regards

Charlie.

[Amal_Antony3331]

Hi @kunal_003 

I'm also using IMX219 sensor with IMX8MP. But facing some issues during integration.

Is your porting IMX219 with IMX8MP is completely successful?

Regards

Charles

[jgsandom]

Hi @Amal_Antony3331,

Can you share specific details on the issues you are facing with the integration?

Thanks,

Joe

[Amal_Antony3331]

Hi @jgsandom 

I'm taking the OV2775 sensor reference for porting the IMX219.

Now able to get the video nodes on v4l2-ctl command.

I'm using the following command to get the stream and attaching the output log.

root@imx8mpevk:/opt/imx8-isp/bin# ./video_test -w 1920 -h 1080 -f RAW10 -t file -m0 -d2
[  506.302106] Enter mipi_csis_clk_enable ...
[  506.306609] Enter mipi_csis_clk_enable ...
[  506.311173] 
[  506.311173]  enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[  506.318604]  Command : VIDIOC_QUERYCAP 
[  506.322901]  enter (( imx219_ioc_qcap ))
[  506.327076]  enter   dwe_runtime_resume  ....
[  506.331451]  enter   dwe_enable_clocks  ....
[  506.335747]  enter   dwe_open ....
[  506.349623]  enter   dwe_close  ....
[  506.365599]  enter   dwe_runtime_suspend  ....
[  506.378250]  Enter mipi_csis_clk_enable ...
[  506.382773]  Enter mipi_csis_clk_enable ...
[  506.387275] 
[  506.387275]  enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[  506.394742]  Command : VIDIOC_QUERYCAP 
[  506.399052]  enter (( imx219_ioc_qcap ))
[  506.403154]  Enter mipi_csis_clk_enable ...
loading [IMX219.drv]...
[  506.427968]  Enter mipi_csis_clk_enable ...
[  506.434057]  Enter mipi_csis_clk_enable ...
[  506.438550] 
[  506.438550]  enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[  506.445963]  Command : VIDIOC_QUERYCAP 
[  506.450243]  enter (( imx219_ioc_qcap ))
[  506.454310]  enter   dwe_runtime_resume  ....
[  506.458677]  enter   dwe_enable_clocks  ....
[  506.462967]  enter   dwe_open ....
 		IMX219_IsiQuerySensorSupportIss (enter)
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev0 0 
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev1 1 
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev2 2 
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev3 3
 
loading [IMX219.drv]...
		IMX219_IsiQuerySensorSupportIss (enter)
Video Test:caps supports:{
        count = 1
        {
        index    = 0
        width    = 1920
        height   = 1080
        fps      = 30
        hdr_mode = 0
        }
}
   
[  506.516725]  enter imx219_priv_ioctl ((( (cmd = 290))))
[  506.525407]  Command : VVSENSORIOC_G_SENSOR_MODE 
[  506.530563]  enter (( imx219_g_mode ))
[  506.534435]  sensor->cur_mode.index = 0 pmode->index = 0 sensor->cur_mode.width =1920 sensor->cur_mode.height =1080 

loading [IMX219.drv]...
 		IMX219_IsiQuerySensorSupportIss (enter)
loading [IMX219.drv]...
 		IMX219_IsiQuerySensorSupportIss (enter)
 		IMX219_IsiGetSensorIss ((enter))
 		IMX219_IsiGetSensorIss ((exit))
 		IMX219_IsiQuerySensorSupportIss (enter)
 		IMX219_IsiCreateSensorIss (enter)
 		IMX219_IsiCreateSensorIss (exit)
 		IMX219_IsiInitSensorIss (enter)
[  506.584657] enter isp_mi_stop
 		IMX219_IsiInitSensorIss (exit)
 		IMX219_IsiGetSensorModeIss (enter)
 		IMX219_IsiGetCapsIss (enter)
 		Bus width == 10 
 		IMX219_IsiGetCapsIss (exit)
isp  input: 1920x1080 RAW10
isp output: 1920x1080 YUV422I
dewarp   input: 1920x1080 YUV422I
dewarp  output: 1920x1080 YUV422I
 		IMX219_IsiSetupSensorIss (enter)
 		IMX219_IsiSetupSensorIss IMX219 System-Reset executed 
[  506.691954] 
[  506.691954]  enter imx219_priv_ioctl ((( (cmd = 283))))
[  506.698675]  Command : VVSENSORIOC_S_HDR_MODE 
[  506.703596]  enter (( imx219_s_hdr ))
[  506.707375]  HDR Selected = 0 
[  506.710721] 
[  506.710721]  enter imx219_priv_ioctl ((( (cmd = 294))))
[  506.717444]  Command : VVSENSORIOC_S_SENSOR_MODE 
[  506.722692]  enter (( imx219_s_sensor_mode ))
[  506.727150]  Mode Selected = 0 
[  506.730309]  enter (( imx219_set_fmt ))
[  506.734241]  enter (( imx219_write_reg_arry ))
[  506.738779]  enter imx219_write_reg_arry
 		 IMX219_AecSetModeParameters: (enter)
		 IMX219_AecSetModeParameters: AecMaxIntegrationTime = 0.033298 
 		 IMX219_AecSetModeParameters: (exit)
 		 IMX219_Private_SetFormat: 1920 1080 0
[  506.805845]   enter (( imx219_get_format_code ))
[  506.810514]   Bayer pattern selected == 2 
 		 IMX219_IsiSetupSensorIss (exit[  506.814723] 
[  506.814723]  enter imx219_priv_ioctl ((( (cmd = 264))))
[  506.824382]  Command :  VVSENSORIOC_WRITE_REG 
[  506.829280]   enter (( imx219_write_reg ))
[  506.833388]  Write reg : reg=601, val=3 

		 IMX219_IsiActivateTestPatte[  506.839040] 
[  506.839040]  enter imx219_priv_ioctl ((( (cmd = 285))))
[  506.847093]  Command : VVSENSORIOC_S_FPS 
[  506.851561]   enter (( imx219_s_fps ))
[  506.855339]  FPS SET = 30 
 		IMX219_IsiRegisterWriteIss (enter)
 [  506.858425] 
[  506.858425]  enter imx219_priv_ioctl ((( (cmd = 290))))
[  506.867980]  Command : VVSENSORIOC_G_SENSOR_MODE 
[  506.873133]  enter (( imx219_g_mode ))
[  506.876981]  sensor->cur_mode.index = 0 pmode->index = 0 sensor->cur_mode.width =1920 sensor->cur_mode.height =1080 

 		IMX219_IsiRe[  506.887801] 
[  506.887801]  enter imx219_priv_ioctl ((( (cmd = 286))))
[  506.897368]  Command : VVSENSORIOC_G_FPS 
[  506.901829]  enter (( imx219_g_fps ))
[  506.905586]  GET FPS = 30 
[  506.908345]  enter imx219_priv_ioctl ((( (cmd = 276))))
[  506.915058]  Command : VVSENSORIOC_S_EXP 
[  506.919515]  enter (( imx219_s_exp ))
[  506.923274]  enter (( imx219_write_reg ))
[  506.927383]  Write reg : reg=15a, val=ff 
[  506.932849]  (exit) result = 0
[  506.932849]  enter imx219_priv_ioctl ((( (cmd = 279))))
[  506.941179]  Command : VVSENSORIOC_S_GAIN 
[  506.945736]  enter (( imx219_s_gain ))
[  506.949581]  enter (( imx219_write_reg ))
[  506.953688]  Write reg : reg=157, val=c8 

 		IMX219_IsiActivateTestPattern[  506.959259] .. Enter mipi_csis_clk_enable ...
[  506.965207] Enter csis_s_fmt ...
[  506.968727] Enter find_csis_format ...
[  506.972751] Enter disp_mix_gasket_config ...
[  506.977297] Enter mipi_csis_set_params ...
[  506.981662] Enter __mipi_csis_set_format ...
[  506.986201] fmt: 0x2008,   1920 x 1080
[  506.990305] Enter mipi_csis_set_hsync_settle ...
[  506.995247] Enter mipi_csis_clk_enable ...
[  506.999780] enter isp_s_comp
[  507.002682] enter isp_s_comp
[  507.005564] enter isp_s_comp
[  507.008514] Enter mipi_csis_clk_enable ...
[  507.012949] Enter mipi_csis_s_stream ...
[  507.017157]  
[  507.017157] Enable = 1 state = 0x0
[  507.022137] Enter mipi_csis_sw_reset ...
[  507.026354] Enter mipi_csis_start_stream ...
[  507.030895] Enter disp_mix_gasket_config ...
[  507.035440] Enter mipi_csis_set_params ...
[  507.039810] Enter __mipi_csis_set_format ...
[  507.044350] fmt: 0x2008,   1920 x 1080
[  507.048460] Enter mipi_csis_set_hsync_settle ...
[  507.053363] Enter mipi_csis_system_enable ...
[  507.057997] Enter disp_mix_gasket_enable ...
[  507.062543] 
[  507.062543]  Enter mipi_csis_clean_irq ...
[  507.068123] 
[  507.068123]  Enter mipi_csis_enable_interrupts ...

 		IMX219_IsiSetSensorFpsIss (enter)
		IMX219_IsiSetSensorFpsIss: set fps(0) out of range, correct to 0 (0, 0)
 		IMX219_IsiSetSensorFpsIss set sensor fps = 0
 		IMX219_IsiSetSensorFpsIss (exit)
 		IMX219_IsiGetSensorModeIss (enter)
[  507.094680] enter wdr3_hw_init
 		IMX219_IsiGetGainLimitsIss (enter)
 [  507.098550] wdr3 res: 1920 1080 
		Min Gain = 1.000000 Max Gain = 21.000000
[  507.105342] enter isp_mi_start
[  507.111282] 
[  507.111282]  enter imx219_priv_ioctl ((( (cmd = 274))))
[  507.120137]  >>>  Command : VVSENSORIOC_S_STREAM 
		IMX219_IsiGetGainLimitsIss (exit)
[  507.124865]  enter (( imx219_s_stream )) control value = 1
 		IMX219_IsiGetIntegrationTimeLimitsIss (enter)
[  507.133292]  enter (( imx219_write_reg ))
[  507.140257]  Write reg : reg=100, val=1 
		Min INtegration Time = 0.000030 Max Int time = 0.033298 
 		IMX219_IsiGetIntegrationTimeLimitsIss (exit)
 		IMX219_IsiGetSensorFpsIss (enter)
		IMX219_IsiGetSensorFpsIss (exit)
 		IMX219_IsiExposureControlIss (enter)
		IMX219_IsiExposureControlIss: g=1.000000, Ti=0.000030
		==>> HDR Mode = 0 
 		IMX219_IsiSetIntegrationTimeIss (enter)
		IMX219_IsiSetIntegrationTimeIss: set AEC_PK_EXPO=0x00001
 		IMX219_IsiSetIntegrationTimeIss  Ti=0.000030
		IMX219_IsiExposureControlIss: set: NewGain=1.000000, hcgTi=0.000030, hcgskip=1
 		IMX219_IsiSetGainIss (enter)
 		IMX219_IsiSetGainIss g=1.000000
		IMX219_IsiExposureControlIss: set: NewGain=1.000000, hcgTi=0.000030, hcgskip=1
		IMX219_IsiExposureControlIss: set: vsg=1.000000, vsTi=0.000030, vsskip=1
 		IMX219_IsiExposureControlIss (exit)
		#### CsiSetFormat 1920 1080 808534599
 		IMX219_IsiGetSensorModeIss (enter)
 		IMX219_IsiGetSensorModeIss (enter)
		CamEngineEnableCproc: set val  1 1 1 1.10 -15 1.00 0.00
 		IMX219_IsiSensorSetStreamingIss (enter)
 		IMX219_IsiSensorSetStreamingIss (exit)

As I can see, till Start Streaming API  called and proper value is written to the register.

But there is no streaming.

./video_test -w 1920 -h 1080 -f RAW10 -t file -m0 -d2 is the command used.

I'm using the Ov2775 xml file for IMX219. Will this cause an issue?

Also can you point out the area where i can look into?

Thanks and Regards

Charles

[joanxie]

this is internal material, I couldn't share here(public community), but I copy the steps here

•Sample code to port new camera Module.

   https://source.codeaurora.org/external/imx/isp-vvcam/

[kunal_003]

Hi @joanxie,

I saw the link(isp-vvacm) provided by you, however I still have below query/confusion. Please help me in this case:

1. From what I understand we need to compile this and install the module by running "insmod.sh". So, it will add userspace support for using ISP. Is this understanding correct?
2. I saw this camera sensors directory, so for SONY IMX219 we need to add support for using ISP?
3. Can we use ISP only by enabling in device tree?
4. Any example pipleline or utility to use streaming via ISP?
5. I am still with many confusions on what exactly needs to be done in order to use ISP. Any documentation/guidance/explanation will be helpful.

Regards,

Kunal

[joanxie]

current bsp doesn't support SONY IMX219  directly, you need porting by yourself, current bsp includes ISP source code directly, you can find them under path after you building the bsp

"imx-yocto-bsp/build-xwayland/tmp/work/imx8mpevk-poky-linux/kernel-module-isp-vvcam/4.2.2.6.1-r0/git/vvcam"

you also can find the source code from:

"https://source.codeaurora.org/external/imx/isp-vvcam/tree/vvcam?h=imx_5.4.70_2.3.0"