[iMX8MP] ISP new sensor porting guide

[imsOPX]

Hi,

I'm in process of research as we are intending to test a few different sensors with iMX8MP and choose the right one for our project based on this SoC. I've choose ov5640 to port and test it first as it is available for the hardware we use - it is capable to output 10 bit RAW over MIPI, so it seems to be good choice. I do NOT want to use the integrated camera ISP as the next sensor to port doesn't have such option.

If this is all successful I can continue with the other sensors as option.

Then I realized the porting guide (i.MX 8M Plus Camera and Display Guide, 2022, Chapter 2) is kind of not very useful as reference what and where to change to adapt different sensor, even if it seems so at first. Also there are some files mentioned in this guide that not actually exist in the isp-imx package, like:

2.5.3 Sensor Driver API
V4l2 Sensor Driver API is declared in file <sensor>_mipi_v3.c , where <sensor> is the name of the sensor (for example, OV2775).

Basically all the topics I found in this forum recommends to create a copy of the existing and supported ov2775 files, adjust the code and modify the appropriate files. I've continued in this direction, using isp-imx-4.2.2.15.0 and yocto build system. I should notice that I'm new to yocto (but old buildroot user ) and also new to iMX (but old FPGA SoC developer), so excuse me if I'm asking too easy questions as newbie here.

Anyway, the list of the modified files and directories is:

./units/isi/drv/OV5640
./units/isi/drv/OV5640/source/OV5640.c
./units/isi/drv/OV5640/calib/OV5640
./units/isi/drv/OV5640/calib/OV5640/OV5640.xml
./units/isi/drv/Sensor1_Entry.cfg
./units/isi/drv/Sensor0_Entry.cfg
./units/isi/drv/OV5640/CMakeLists.txt
./vvcam/v4l2/sensor/ov5640
./vvcam/v4l2/sensor/ov5640/ov5640_regs_1080p.h
./mediacontrol/server/CMakeLists.txt
./appshell/shell_libs/CMakeLists.txt
./imx/run.sh
./imx/start_isp.sh

If I understand correctly this will produce the desired .drv file and calibration .xml (which will be dummy/incorrect), the lib files, etc. and will populate the rootfs with them. The calibration will be something to worry at later stage. I will change the appropriate dtb file to support the sensor on one of the available MIPI CSI ports later. Please, let me know if I'm missing something here?

Right now the isp-imx is compiled correctly. But at package_split stage of this package processing the yocto build is returning error related to the position of libov5640.so - it should be in dev, but isn't. So I'm stuck here.

The error output of bitbake is as follow:

ERROR: isp-imx-4.2.2.15.0-r0 do_package_qa: QA Issue: non -dev/-dbg/nativesdk- package isp-imx contains symlink .so '/usr/lib/libov5640.so' [dev-so]
ERROR: isp-imx-4.2.2.15.0-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.
ERROR: Logfile of failure stored in: /workdir/build_wayland/tmp/work/cortexa53-crypto-mx8mp-poky-linux/isp-imx/4.2.2.15.0-r0/temp/log.do_package_qa.18024
ERROR: Task (/workdir/sources/meta-optix/recipes-bsp/isp-imx/isp-imx_4.2.2.15.0.bb:do_package_qa) failed with exit code '1'

The libov****.so files at this point in /tmp/work/...../isp-imx/4.2.2.15.0-r0:

./packages-split/isp-imx/usr/lib/libov2775.so.1.0.0
./packages-split/isp-imx/usr/lib/libov5640.so.1.0.0
./packages-split/isp-imx/usr/lib/libov2775.so.1
./packages-split/isp-imx/usr/lib/libov5640.so.1
./packages-split/isp-imx/usr/lib/libov5640.so
./packages-split/isp-imx-dbg/usr/lib/.debug/libov2775.so.1.0.0
./packages-split/isp-imx-dbg/usr/lib/.debug/libov5640.so.1.0.0
./packages-split/isp-imx-dev/usr/lib/libov2775.so

I'm not sure is it the yocto inducing the problem or I haven't done something properly in the adding the sensor in isp-imx and this is the result?

Some help here, please?

It will be really helpful if someone has added new sensor to isp-imx and is willing to share some repo and/or steps how it is done - it will help all with the same problem to track down their mistakes. I'm quite sure I have done some fatal mistakes until now, which will be fun to debug

[ederibaucourt]

Hello @imsOPX , I would like to use the OV5640 with the ISP. Would it be possible for you to share all the files you mention regarding porting this camera for the ISP? Even if they are not fully working, I'd be very interested.

Best Regards,

Enguerrand de Ribaucourt from Savoir-faire Linux

[imsOPX]

Hi @ederibaucourt ,

I'm sorry but it is not possible to share the files. Even if they were only for training (before porting the real sensors) - they are still property of my employer. In fact the sensor was working quite well with the basic drivers I have written for the ISP. If you have some specific questions and if I may help you - I will try, just post me a PM.
Have a nice day!

[imsOPX]

At the end I have the sensor up and running using the IMX8M Plus ISP and the result is quite acceptable. The problems were mostly my errors in the kernel driver module. Well, there is some problem with the AEC feature of the ISP, but I will start another topic for this.

[chp0423]

@imsOPX In your previous posts you said the gst-launch presented no errors but no images. How did you solve this issue? I have met the similar problem with you. Thanks sincerely.

[Amal_Antony3331]

 Hi @imsOPX 

It would be really helpful if you can provide some points to my issue.

Im trying to integrate IMX219 with IMX8MP.

Now I'm able to get the video nodes  in the v4l2-ctl command

root@imx8mpevk:/opt/imx8-isp/bin# v4l2-ctl --list-devices

 ():
        /dev/v4l-subdev0
        /dev/v4l-subdev3
        /dev/v4l-subdev4

 ():
        /dev/v4l-subdev1
        /dev/v4l-subdev5
        /dev/v4l-subdev6

 (csi0):
        /dev/v4l-subdev2

FSL Capture Media Device (platform:mxc-md):
        /dev/media0

VIV (platform:viv0):
        /dev/video2

VIV (platform:viv1):
        /dev/video3

vsi_v4l2dec (platform:vsi_v4l2dec):
        /dev/video1

vsi_v4l2enc (platform:vsi_v4l2enc):
        /dev/video0

viv_media (platform:vvcam-video):
        /dev/media1

root@imx8mpevk:/opt/imx8-isp/bin# 

  But when i try to run the gstreamer command , getting the following error

ndsinkmx8mpevk:/opt/imx8-isp/bin# gst-launch-1.0 -v v4l2src device=/dev/video2 ! "video/x-raw,format=YUY2,width=1920,height=1080, framerate=30/1" ! queue ! imxvideoconvert_g2d ! waylandsink
[ 2473.477604]    enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[ 2473.484168]  <<<  Command : VIDIOC_QUERYCAP 
[ 2473.488464]   enter (( imx219_ioc_qcap ))
[ 2473.885333]    enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[ 2473.892340]  <<<  Command : VIDIOC_QUERYCAP 
[ 2473.896640]   enter (( imx219_ioc_qcap ))
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 2474.301297]    enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[ 2474.304047] ... Return Value = 0 ..... 
[ 2474.310120]  <<<  Command : VIDIOC_QUERYCAP 
[ 2474.310124]   enter (( imx219_ioc_qcap ))
[ 2474.489256]    enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[ 2474.493931] ... Return Value = 0 ..... 
[ 2474.500042]  <<<  Command : VIDIOC_QUERYCAP 
[ 2474.508151]   enter (( imx219_ioc_qcap )) 
loading [IMX219.drv]...
[ 2474.593452]    enter imx219_priv_ioctl ((( (cmd = 289))))
[ 2474.598881]  <<<  Command : VVSENSORIOC_QUERY 
[ 2474.603443]   enter (( imx219_ioc_query_mode ))
ERROR  : [MediaPipeline] NativeSensor open error!
ERROR  : [V4l2Event] initialize MediaPipeline error!

[ 2479.485515] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 2479.645507] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 2479.809525] viv_post_event: unsubscribed event id =14 type=0x08002000
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fract9
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=9
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction9
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio9
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)9
/GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fractx
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, intx
/GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoco[ 2479.973514] viv_post_event: unsubscribed event id =14 type=0x08002000
nvert_g2d0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string9
[ 2480.149502] viv_post_event: unsubscribed event id =7 type=0x08002000
[ 2480.313526] viv_post_event: unsubscribed event id =6 type=0x08002000
[ 2480.477526] viv_post_event: unsubscribed event id =10 type=0x08002000
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate required memory.
Additional debug info:
../git/sys/v4l2/gstv4l2src.c(659): gst_v4l2src_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Buffer pool activation failed
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../git/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:06.475932375
Setting pipeline to NULL ...
[ 2480.653516] viv_post_event: unsubscribed event id =1 type=0x08002000
Freeing pipeline ...
root@imx8mpevk:/opt/imx8-isp/bin# 

I think, we will get more debug info from isp-imx if we are able to enable the trace .

Can you please guide on this.

Best Regards

Charlie

[imsOPX]

Hi @Amal_Antony3331 

actually I wasn't able to enable the TRACE globally (and I'm sure it will flood you with A LOT  of information). So I did something simpler instead:

// ---> Start of CREATE_TRACER
enum{   IMX_VERBOSE = 0,
        IMX_INFO,
        IMX_WARN,
        IMX_ERROR,
        IMX_QUIET
    };

#define IMX_DEBUG_LEVEL IMX_ERROR

#undef TRACE
#define TRACE(x, fmt, ...)                          \
            do {                                    \
                if( x >= IMX_DEBUG_LEVEL )       \
                    printf(fmt, ##__VA_ARGS__);     \
            } while(0)                              \
// ---> End of CREATE_TRACER

....
TRACE( IMX_INFO, "%s: (enter)\n", __func__);
....

Well, it isn't very good solution, but it does the job - controlling the level of debug I'm sorry, I can't help you, may be @khang_letruong will shed some light on to the problem, I think he was successfuly enabled the TRACE? It may be helpful for all of us!

[imsOPX]

@khang_letruongThanks for the support, I appreciate it. So far I've been able to execute start_isp.sh without errors, the log is as follows.

/opt/imx8-isp/bin# ./start_isp.sh 
Starting isp_media_server with OV5640 
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "ov5640"...
Removing vvcam-isp...
[  142.597453] enter viv_isp_exit_module
[  142.601271] enter isp_hw_remove
[  142.604439] vvcam isp driver removed
Removing vvcam-dwe...
[  142.641720] enter viv_dwe_exit_module
[  142.645517] enter dwe_hw_remove
[  142.648798] enter fake_pdev_release
[  142.652319] vvcam dewarp driver removed
Removing vvcam-video...
Removing imx8-media-dev...
[  142.706337] : Unregistered all entities
Removing ov5640...
[  142.752475] enter ov5640_remove
[  142.755626] ov5640 power off
[  142.758829] enter ov5640_regulator_disable
[  142.763409] enter ov5640_remove
[  142.766595] ov5640 power off
[  142.769964] enter ov5640_regulator_disable
Removing ov2775...
Removing os08a20...
Removing basler-camera-driver-vvcam...
Loading module ov5640 ...
ov5640
[  142.923721] enter ov5640_probe
[  142.927950] enter ov5640_retrieve_capture_properties
[  142.932955] ov5640 1-003c: supply DOVDD not found, using dummy regulator
[  142.939948] ov5640 1-003c: supply DVDD not found, using dummy regulator
[  142.946767] ov5640 1-003c: supply AVDD not found, using dummy regulator
[  142.953459] enter ov5640_set_clk_rate
[  142.957161] ov5640 power on
[  142.960239] ov5640 resetting
[  143.048885] Chip ID = 0x5640
[  143.051781] ov5640_probe camera mipi ov5640, is found
[  143.057043] enter ov5640_probe
[  143.061194] enter ov5640_retrieve_capture_properties
[  143.066198] ov5640 3-003c: supply DOVDD not found, using dummy regulator
[  143.073060] ov5640 3-003c: supply DVDD not found, using dummy regulator
[  143.079780] ov5640 3-003c: supply AVDD not found, using dummy regulator
[  143.086499] enter ov5640_set_clk_rate
[  143.090197] ov5640 power on
[  143.093276] ov5640 resetting
[  143.184881] Chip ID = 0x5640
[  143.187777] ov5640_probe camera mipi ov5640, is found
Loaded /lib/modules/5.10.52+gf455d15ccde8/extra/sensor/ov5640/ov5640.ko 
Loading module imx8-media-dev ...
imx8_media_dev
[  143.224037] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[  143.234847] mx8-img-md: Registered sensor subdevice: ov5640 3-003c (1)
[  143.241455] mx8-img-md: Registered sensor subdevice: ov5640 1-003c (2)
[  143.248057] enter ov5640_link_setup
[  143.251558] mx8-img-md: created link [ov5640 1-003c] => [mxc-mipi-csi2.0]
[  143.258371] enter ov5640_link_setup
[  143.261885] mx8-img-md: created link [ov5640 3-003c] => [mxc-mipi-csi2.1]
[  143.268694] mxc-md 32c00000.bus:camera: mxc_md_create_links
[  143.277847] ov5640_priv_ioctl: CMD = 0x80685600
[  143.280641] ov5640_priv_ioctl: CMD = 0x80685600
[  143.285231] enter ov5640_query_capability
Loaded /lib/modules/5.10.52+gf455[  143.286962] enter ov5640_query_capability
d15ccde8/kernel/drivers/staging/media/imx/imx8-media-dev.ko 
Loading module vvcam-video ...
vvcam_video
[  143.329094] enter viv_video_init_module
Loaded /lib/modules/5.10.52+gf455d15ccde8/extra/video/vvcam-video.ko 
Loading module vvcam-dwe ...
vvcam_dwe
[  143.366613] enter viv_dwe_init_module
[  143.370573] enter dwe_hw_probe
[  143.373957] vvcam dewarp driver probed
Loaded /lib/modules/5.10.52+gf455d15ccde8/extra/vvcam-dwe.ko 
Loading module vvcam-isp ...
vvcam_isp
[  143.409671] enter viv_isp_init_module
[  143.413712] enter isp_hw_probe
[  143.417172] vvcam isp driver registered
Loaded /lib/modules/5.10.52+gf455d15ccde8/extra/vvcam-isp.ko 
Starting isp_media_server with c[  143.428030] enter isp_mi_stop
onfiguration file CAMERA0

When I try to play the video stream using gstreamer there are no errors. The log is:

/opt/imx8-isp/bin# gst-launch-1.0 -v v4l2src device=/dev/video0 ! "video/x-raw,format=YUY2,width=3840,height=2160" ! queue ! imxvideoconvert_g2d ! waylandsink                         
[  363.172878] ov5640_priv_ioctl: CMD = 0x80685600
[  363.177423] enter ov5640_query_capability
[  363.181688] ov5640_priv_ioctl: CMD = 0x80685600
[  363.186257] enter ov5640_query_capability
[  363.200339] enter isp_mi_stop
Setting pipeline to PAUSED ...
[  363.269814] ov5640_priv_ioctl: CMD = 0x80685600
Pipeline is live and does not nee[  363.274821] enter ov5640_query_capability
d PREROLL ...
Pipeline is PREROL[  363.281727] ov5640_priv_ioctl: CMD = 0x80685600
LED ...
Setting pipeline to PLAY[  363.289094] enter ov5640_query_capability
ING ...
New clock: GstSystemClock
[  363.340146] ov5640_priv_ioctl: CMD = 0x80685600
[  363.344703] enter ov5640_query_capability
[  363.348798] ov5640_priv_ioctl: CMD = 0x80685600
[  363.353375] enter ov5640_query_capability
loading [ov5640.drv]...
OV5640_IsiHalQuerySensorIss (en[  363.369883] ov5640_priv_ioctl: CMD = 0x105
ter) 
[  363.376855] enter ov5640_query_supports
OV5640_IsiHalQuerySensorIss (exit[  363.381900] ov5640_priv_ioctl: CMD = 0x80685600
)
WARN   : [DEWARP_HW][  363.388755] enter ov5640_query_capability
 dwe type /dev/v4l-subdev0 0 [  363.395665] ov5640_priv_ioctl: CMD = 0x80685600

WARN   : [DEWARP_HW] [  363.403047] enter ov5640_query_capability
dwe type /dev/v4l-subdev1 1 
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev2 2 
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev3 3 
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev4 4 
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mo1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interl1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=[  363.527092] ov5640_priv_ioctl: CMD = 0x105
(int)3840, height=(int)2160, fram[  363.532622] enter ov5640_query_supports
erate=(fraction)30/1, pixel-aspec[  363.539386] ov5640_priv_ioctl: CMD = 0x101
t-ratio=(fraction)1/1, interlace-[  363.546333] ov5640_priv_ioctl: CMD = 0x104
mode=(string)progressive, colorim[  363.553299] ov5640_priv_ioctl: CMD = 0x103
etry=(string)2:6:11:1
/GstPipeli[  363.560261] ov5640_priv_ioctl: CMD = 0x100
ne:pipeline0/imxvideoconvert_g2d:[  363.567218] ov5640_priv_ioctl: CMD = 0x106
imxvideoconvert_g2d0.GstPad:src: [  363.574205] enter ov5640_set_sensor_mode
caps = video/x-raw, width=(int)38[  363.580987] ov5640_priv_ioctl: CMD = 0x107
40, height=(int)2160, framerate=([  363.587950] enter ov5640_get_sensor_mode
fraction)30/1, pixel-aspect-ratio[  363.594785] ov5640_priv_ioctl: CMD = 0x105
=(fraction)1/1, interlace-mode=(s[  363.601720] enter ov5640_query_supports
tring)progressive, colorimetry=(s[  363.609673] enter isp_mi_stop
tring)2:6:11:1, format=(string)BG[  363.616828] enter ov5640_set_fmt
Rx
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = video/x-raw, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)px
/GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, [  363.652933] ov5640_set_fmt: 1920x1080
height=(int)2160, framerate=(frac[  363.659203] enter ov5640_get_format_code
tion)30/1, pixel-aspect-ratio=(fr[  363.665988] ov5640_get_format_code: pattern = 0x3 | bit_width = 0xa
action)1/1, interlace-mode=(strin[  363.675157] ov5640_priv_ioctl: CMD = 0x11d
g)progressive, colorimetry=(strin[  363.682066] enter ov5640_set_test_pattern
g)2:6:11:1
isp  input: 1920x1080[  363.689249] ov5640_priv_ioctl: CMD = 0x117
 RAW10
isp output: 1920x1080 YUV[  363.695864] enter ov5640_set_fps
422I
dewarp   input: 1920x1080 Y[  363.702518] ov5640_priv_ioctl: CMD = 0x107
UV422I
dewarp  output: 3840x2160[  363.708915] enter ov5640_get_sensor_mode
 YUV422I
OV5640_IsiGetSensorIss [  363.716046] ov5640_priv_ioctl: CMD = 0x112
(enter)
OV5640_IsiGetSensorIss ([  363.722635] enter ov5640_set_exp
exit)
OV5640_IsiHalQuerySensorIs[  363.730392] ov5640_priv_ioctl: CMD = 0x115
s (enter) 
OV5640_IsiHalQuerySen[  363.735669] enter ov5640_set_gain
sorIss (exit)
OV5640_IsiCreateSe[  363.743553] enter isp_s_comp
nsorIss (enter)
OV5640_IsiSensor[  363.747610] enter isp_s_comp
SetPowerIss: (enter)
OV5640_IsiS[  363.753350] enter isp_s_comp
ensorSetPowerIss: set power 1
OV5640_IsiSensorSetPowerIss: (exit)
OV5640_IsiSensorGetClkIss: (enter)
OV5640_IsiSensorGetClkIss: status:276050132 sensor_mc[  363.772367] enter wdr3_hw_init
lk:24000000 csi_max_pixel_clk:500[  363.775730] wdr3 res: 1920 1080 
000000
OV5640_IsiSensorGetClkIss[  363.782417] enter isp_mi_start
: (exit)
OV5640_IsiSensorSetClkI[  363.788205] ov5640_priv_ioctl: CMD = 0x110
ss: (enter)
OV5640_IsiSensorSetC[  363.794727] ov5640_s_stream: 1
lkIss: status:1 sensor_mclk:24000000 csi_max_pixel_clk:500000000
OV5640_IsiSensorSetClkIss: (exit)
OV5640_IsiResetSensorIss: (enter)
OV5640_IsiResetSensorIss: (exit)
OV5640_IsiSetSensorModeIss (enter)
OV5640_IsiSetSensorModeIss (exit) 
OV5640_IsiCreateSensorIss (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit) 
OV5640_IsiGetCapsIss (enter) 
OV5640_IsiQuerySensorIss (enter) 
OV5640_IsiHalQuerySensorIss (enter) 
OV5640_IsiHalQuerySensorIss (exit)
OV5640_IsiQuerySensorIss (exit)
OV5640_IsiGetCapsIss (exit)
OV5640_IsiSetupSensorIss (enter)
OV5640_IsiSetupSensorIss (exit)
OV5640_IsiSetTestPatternIss (enter)
OV5640_IsiSetTestPatternIss: test pattern enable[0] mode[0]
OV5640_IsiSetTestPatternIss: (exit)
OV5640_IsiSetSensorFpsIss: (enter)
OV5640_IsiSetSensorFpsIss: (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit) 
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit) 
OV5640_IsiGetAeInfoIss (enter)
OV5640_IsiGetAeInfoIss (exit)
OV5640_IsiGetSensorFpsIss: (enter)
OV5640_IsiGetSensorFpsIss: (exit)
OV5640_IsiGetAeStartExposureIs (enter)
OV5640_IsiGetAeStartExposureIs:get start exposure 36403200
OV5640_IsiGetAeStartExposureIs: (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit) 
OV5640_IsiGetAeInfoIss (enter)
OV5640_IsiGetAeInfoIss (exit)
OV5640_IsiSetIntegrationTimeIss (enter)
OV5640_IsiSetIntegrationTimeIss set linear exp 400 
OV5640_IsiSetIntegrationTimeIss (exit)
OV5640_IsiSetGainIss (enter)
OV5640_IsiSetGainIss set linear gain 3072
OV5640_IsiSetGainIss (exit)
OV5640_IsiGetIntegrationTimeIss (enter)
OV5640_IsiGetIntegrationTimeIss (exit)
OV5640_IsiGetGainIss (enter)
OV5640_IsiGetGainIss (exit)
#### CsiSetFormat 1920 1080 808535874
OV5640_IsiGetSensorIspStatusIss: (enter)
OV5640_IsiGetSensorIspStatusIss: (exit)
OV5640_IsiGetSensorIspStatusIss: (enter)
OV5640_IsiGetSensorIspStatusIss: (exit)
OV5640_IsiGetSensorIspStatusIss: (enter)
OV5640_IsiGetSensorIspStatusIss: (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit) 
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit) 
OV5640_IsiGetSensorIspStatusIss: (enter)
OV5640_IsiGetSensorIspStatusIss: (exit)
CamEngineEnableCproc: set val  1 1 1 1.10 -15 1.00 0.00
OV5640_IsiSensorSetStreamingIss (enter)
OV5640_IsiSensorSetStreamingIss: set streaming 1
OV5640_IsiSensorSetStreamingIss (exit) 
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:08.579723250
Setting pipeline to NULL ...
[  371.891324] enter isp_mi_stop
OV5640_IsiSensorSetStreamingIss ([  371.997447] ov5640_priv_ioctl: CMD = 0x110
enter)
[  372.004419] ov5640_s_stream: 0
OV5640_IsiSensorSetStreamingIss: set streaming 0
OV5640_IsiSensorSetStreamingIss (exit) 
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit) 
OV5640_IsiReleaseSensorIss (enter[  372.024207] ov5640_priv_ioctl: CMD = 0x110
) 
OV5640_IsiSensorSetStreamingI[  372.031192] ov5640_s_stream: 0
ss (enter)
[  372.037709] ov5640_priv_ioctl: CMD = 0x104
OV5640_IsiSensorSetStreamingIss: [  372.042286] ov5640_priv_ioctl: CMD = 0x103
set streaming 0
OV5640_IsiSensor[  372.049251] ov5640_priv_ioctl: CMD = 0x101
SetStreamingIss (exit) 
OV5640_IsiSensorGetClkIss: (enter)
OV5[  372.060986] enter isp_mi_stop
640_IsiSensorGetClkIss: status:276050132 sensor_mclk:24000000 csi_max_pixel_clk:500000000
OV5640_IsiSensorGetClkIss: (exit)
OV5640_IsiSensorSetClkIss: (enter)
OV5640_IsiSensorSetClkIss: status:0 sensor_mclk:24000000 csi_max_pixel_clk:500000000
OV5640_IsiSensorSetClkIss: (exit)
OV5640_IsiSensorSetPowerIss: (enter)
OV5640_IsiSensorSetPowerIss: set power 0
OV5640_IsiSensorSetPowerIss: (exit)
OV5640_IsiReleaseSensorIss (exit)
Freeing pipeline ...

The logs contains some debug prints from my driver as you see, just to be sure the proper functions are called.

Well, so far there is no image.. and I'm thinking what can be wrong. Most probably my ov5640 driver isn't ok, at least to say. I'm using some values from ov5640_mipi.c linux kenel driver init structures, anyway - there are a lot of registers and it is easy to make mistake

[khang_letruong]

Hi @imsOPX ,

Please share the output of following commands :

$ v4l2-ctl --list-devices
$ v4l2-ctl --device /dev/video0 --all

Regards,
Khang

[imsOPX]

@khang_letruong, here you are:

v4l2-ctl --list-devices

[ 2781.351229] ov5640_priv_ioctl: CMD = 0x80685600
[ 2781.355856] enter ov5640_query_capability
[ 2781.360079] ov5640_priv_ioctl: CMD = 0x80685600
[ 2781.364626] enter ov5640_query_capability
 ():
        /dev/v4l-subdev0
        /dev/v4l-subdev4
        /dev/v4l-subdev5

 (
):
        /dev/v4l-subdev1

 (csi0):
        /dev/v4l-subdev3

 (csi1):
        /dev/v4l-subdev2

FSL Capture Media Device (platform:mxc-md):
        /dev/media0

VIV (platform:viv0):
        /dev/video0

viv_media (platform:vvcam-video):
        /dev/media1

v4l2-ctl --device /dev/video0 --all

Driver Info:
        Driver name      : viv_v4l2_device
        Card type        : VIV
        Bus info         : platform:viv0
        Driver version   : 5.10.52
        Capabilities     : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : vvcam-video
        Model            : viv_media
        Serial           : 
        Bus info         : 
        Media version    : 5.10.52
        Hardware revision: 0x00000000 (0)
        Driver version   : 5.10.52
Interface Info:
        ID               : 0x03000003
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : viv_v4l20
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Sink, Must Connect
          Link 0x02000010: from remote pad 0x1000006 of entity 'vvcam-dwe.0': Data, Enabled
Priority: 2
Video input : 0 (camera: ok)
Format Video Capture:
        Width/Height      : 3840/2160
        Pixel Format      : 'YUYV' (YUYV 4:2:2)
        Field             : None
        Bytes per Line    : 7680
        Size Image        : 16588800
        Colorspace        : Rec. 709
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: BT.2020
        Quantization      : Limited Range
        Flags             : 
Crop Capability Video Capture:
        Bounds      : Left 0, Top 0, Width 3840, Height 2160
        Default     : Left 0, Top 0, Width 3840, Height 2160
        Pixel Aspect: 1/1
Selection Video Capture: crop, Left 0, Top 0, Width 3840, Height 2160, Flags: 
Selection Video Capture: crop_default, Left 0, Top 0, Width 3840, Height 2160, Flags: 
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 3840, Height 2160, Flags: 
Selection Video Capture: compose, Left 0, Top 0, Width 3840, Height 2160, Flags: 
Selection Video Capture: compose_default, Left 0, Top 0, Width 3840, Height 2160, Flags: 
Selection Video Capture: compose_bounds, Left 0, Top 0, Width 3840, Height 2160, Flags: 
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 30.000 (30/1)
        Read buffers     : 0

User Controls

                   viv_ext_ctrl 0x0098f901 (str)    : min=0 max=65535 step=1 value='{\n\x09"dwe" : \n\x09{\n\x09\x09"bypass" : false,\n\x09\x09"hflip" : false,\n\x09\x09"mat" : \n\x09\x09[\n\x09\x09\x096d
WARN   : [V4l2Event] extctrl2: cid=0x98f903,ret=0
                    sensor_resw 0x0098f903 (int)    : min=0 max=3840 step=1 default=1920 value=1920 flags=read-only, volatile
WARN   : [V4l2Event] extctrl2: cid=0x98f904,ret=0
                    sensor_resh 0x0098f904 (int)    : min=0 max=2160 step=1 default=1080 value=1080 flags=read-only, volatile

Now I see there is something wrong with the max height and width - this sensor is capable of 2592x1944, not 3840x2160, anyway I have tried with 1920x1080, it is still the same result - no error, no image.

As I understand you have ported imx327 to isp-imx, right?

[khang_letruong]

Hi @imsOPX ,

Yes, I did port imx296, imx327, ar0234 and other chinese sensors' drivers with 1440x1080 and 1920x1080 resolutions. Currently, I am porting a 4K one.

Regards,
Khang

[imsOPX]

Well, as expected, the concrete problem was aftermath of my mistake in the recipe for isp-imx. Always check your FILES_${PN}-dev += ...

Anyway, as this is only part of the process, I will like to left the topic open for further questions related to the porting process.

[khang_letruong]

Hi @imsOPX ,

I am not sure if you have resolved your issue but there's example of how to integrate custom isp-imx into Yocto here : https://github.com/boundarydevices/meta-boundary/tree/master/recipes-bsp/isp-imx

Also, as for your issue I confirm the solution by adding following lines into isp-imx_%.bbappend as below :

FILES_${PN}-dev += " \
   ${libdir}/lib<custom-sensor-name>.so \ 
"

Regards,

Khang

[imsOPX]

Thank you for your answer, @khang_letruong

Yes, this was the problem, as I mentioned above.

Now I'm trying to understand why it is not working at all. Right now I'm attempting to enable trace prints in my driver as part of the ISP. I already enabled with 1s TRACE( .., .., 1) prints, but nothing shows up/ I'm a bit of confused is the ISP framework calling the correct functions at all? I can see the kernel driver for my sensor up, but the "glue" .drv functions are strangely quiet.

I have disabled the imx8-isp.service and I'm starting it manually with start_isp.sh:

root@imx8mp-var-dart:/opt/imx8-isp/bin# ./start_isp.sh 
Starting isp_media_server with OV5640 Optix
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "optix_ov5640"...
Removing vvcam-isp...
Removing vvcam-dwe...
Removing vvcam-video...
[  917.253749] enter viv_video_exit_module
Removing imx8-media-dev...
[  917.396216] : Unregistered all entities
Removing ov5640...
Removing ov2775...
Removing os08a20...
Removing basler-camera-driver-vvcam...
Loading module ov5640 ...
ov5640
[  917.538486] ov5640 1-003c: supply DOVDD not found, using dummy regulator
[  917.545763] ov5640 1-003c: supply AVDD not found, using dummy regulator
[  917.552521] ov5640 1-003c: supply DVDD not found, using dummy regulator
[  917.601445] ov5640 3-003c: supply DOVDD not found, using dummy regulator
[  917.608443] ov5640 3-003c: supply AVDD not found, using dummy regulator
[  917.615299] ov5640 3-003c: supply DVDD not found, using dummy regulator
Loaded /lib/modules/5.10.52+ga18cf3faaeed/kernel/drivers/media/i2c/ov5640.ko 
Loading module imx8-media-dev ...
imx8_media_dev
[  917.701182] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[  917.712059] mx8-img-md: Registered sensor subdevice: ov5640 3-003c (1)
[  917.718649] mx8-img-md: Registered sensor subdevice: ov5640 1-003c (2)
[  917.725211] mx8-img-md: created link [ov5640 1-003c] => [mxc-mipi-csi2.0]
[  917.732014] mx8-img-md: created link [ov5640 3-003c] => [mxc-mipi-csi2.1]
[  917.738844] mxc-md 32c00000.bus:camera: mxc_md_create_links
Loaded /lib/modules/5.10.52+ga18cf3faaeed/kernel/drivers/staging/media/imx/imx8-media-dev.ko 
Loading module vvcam-video ...
vvcam_video
[  917.777826] enter viv_video_init_module
Loaded /lib/modules/5.10.52+ga18cf3faaeed/extra/video/vvcam-video.ko 
Loading module vvcam-dwe ...
vvcam_dwe
vvcam_dwe              28672  2[  917.799077] enter isp_mi_stop

vvcam-dwe already loaded.
Loading module vvcam-isp ...
vvcam_isp
vvcam_isp              69632  2
vvcam-isp already loaded.
Starting isp_media_server with configuration file CAMERA0
INFO   : [ISP_MEDIA_SERVER] ******************************************************************
INFO   : [ISP_MEDIA_SERVER] VIV ISP Media Control Framework V4.2.2p15 (Aug  1 2022, 13:09:03)
INFO   : [ISP_MEDIA_SERVER] ******************************************************************

And the result of v4l2-ctl --all

root@imx8mp-var-dart:/opt/imx8-isp/bin# v4l2-ctl --all
INFO   : [MediaPipeline] NativeSensor Creat
[ 1015.143134] enter isp_mi_stop
ERROR  : [MediaPipeline] NativeSensor open error!
ERROR  : [V4l2Event] initialize MediaPipeline error!

Driver Info:
        Driver name      : viv_v4l2_device
        Card type        : VIV
        Bus info         : platform:viv0
        Driver version   : 5.10.52
        Capabilities     : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : vvcam-video
        Model            : viv_media
        Serial           : 
        Bus info         : 
        Media version    : 5.10.52
        Hardware revision: 0x00000000 (0)
        Driver version   : 5.10.52
Interface Info:
        ID               : 0x03000003
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : viv_v4l20
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Sink, Must Connect
          Link 0x0200000a: from remote pad 0x1000009 of entity 'vvcam-isp.0': Data, Enabled
Priority: 2
Video input : 0 (camera: ok)
Format Video Capture:
        Width/Height      : 0/0
        Pixel Format      : ''
        Field             : None
        Bytes per Line    : 0
        Size Image        : 0
        Colorspace        : Rec. 709
        Transfer Function : Default (maps to Rec. 709)
        YCbCr/HSV Encoding: Default (maps to Rec. 709)
        Quantization      : Default (maps to Full Range)
        Flags             : 
[ 1020.286862] viv_post_event: unsubscribed event id =14 type=0x08002000
Crop Capability Video Capture:
        Bounds      : Left 0, Top 0, Width 0, Height 0
        Default     : Left 0, Top 0, Width 0, Height 0
        Pixel Aspect: 1/1
[ 1020.446856] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: crop, Left 0, Top 0, Width 0, Height 0, Flags: 
[ 1020.610862] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: crop_default, Left 0, Top 0, Width 0, Height 0, Flags: 
[ 1020.774862] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 0, Height 0, Flags: 
[ 1020.938869] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: compose, Left 0, Top 0, Width 0, Height 0, Flags: 
[ 1021.102863] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: compose_default, Left 0, Top 0, Width 3840, Height 2160, Flags: 
[ 1021.266863] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: compose_bounds, Left 0, Top 0, Width 3840, Height 2160, Flags: 
[ 1021.430862] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 1021.594862] viv_post_event: unsubscribed event id =14 type=0x08002000
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: invalid (0/1)
        Read buffers     : 0

User Controls

                   viv_ext_ctrl 0x0098f901 (str)    : min=0 max=65535 step=1 value='' flags=has-payload
[ 1021.758862] viv_post_event: unsubscribed event id =14 type=0x08002000
error 22 getting ctrl sensor.resw
[ 1021.922863] viv_gen_g_ctrl: unsubscribed event id =11 type=0x08002000
error 22 getting ctrl sensor.resh
[ 1022.086863] viv_gen_g_ctrl: unsubscribed event id =11 type=0x08002000

And I'm stuck here..

[jeho_park]

Hi  imsOPX  

I have two questions.  

First is Why did you port Ov5640 to the isp-imx ?  because the driver of the ov5649 is already ported in the kernel source below

drivers/media/platform/mxc/capture/ov5640_mipi_v2.c

drivers/media/platform/mxc/capture/ov5640_v2.c

drivers/media/i2c/ov5640.c 

Did you have any reason to port the ov5640 sensor to the isp-imx package ??

Second,  I have a custom board whose cpu is imx8mp, there is a ISP and the imx273 image sensor.

This ISP controls the imx273 sensor thru I2C and i assume i can get image stream thru MIPI and V4L2 buffer.

This ISP provides several functionalities and I2C interface to communicate.

As a result, the control path is different with that of the isp-imx. I wonder I have to port this ISP to the isp-imx package of the NXP ?  

Do you think There is a need i have to port the ISP to the isp-imx ?? 

I want to hear of your idea. i will wait your reply. previously thanks 

[Amal_Antony3331]

Hi @imsOPX 

I was trying to port the IMX219 with the IMX8MP. Can you please guide me to enable the traceing in the isp-imx?

Regards

Charlie

[khang_letruong]

Hi again,

For following error:

root@imx8mp-var-dart:/opt/imx8-isp/bin# v4l2-ctl --all
INFO   : [MediaPipeline] NativeSensor Creat
[ 1015.143134] enter isp_mi_stop
ERROR  : [MediaPipeline] NativeSensor open error!
ERROR  : [V4l2Event] initialize MediaPipeline error!

Maybe you should compare the driver name included in your .drv file (or its source <sensor-name>.c such as OV5640.c) and the driver name in the XML file(s) as well.

Regards,
K.

[khang_letruong]

Hi @imsOPX,

I already enabled with 1s TRACE( .., .., 1) prints, but nothing shows up/ I'm a bit of confused is the ISP framework calling the correct functions at all

Should you also need to export the ISP_LOG_LEVEL=8?

According to my experience, in older version (isp-imx-4.2.2.11.0), it was not straight-forward to enable to tracing by just setting 1s to TRACE() macros. I am not sure if they (NXP) have been more "open" on that in recent versions. Also they confirmed to me that the Full build had not been released to customers : https://community.nxp.com/t5/i-MX-Processors/IMX8MP-isp-imx-Undefined-symbol-trace/m-p/1312440#M1773...

I was able to enable to tracing of isp-imx via their professional support channel, however.

Regards,
K.