Hi,
I'm in process of research as we are intending to test a few different sensors with iMX8MP and choose the right one for our project based on this SoC. I've choose ov5640 to port and test it first as it is available for the hardware we use - it is capable to output 10 bit RAW over MIPI, so it seems to be good choice. I do NOT want to use the integrated camera ISP as the next sensor to port doesn't have such option.
If this is all successful I can continue with the other sensors as option.
Then I realized the porting guide (i.MX 8M Plus Camera and Display Guide, 2022, Chapter 2) is kind of not very useful as reference what and where to change to adapt different sensor, even if it seems so at first. Also there are some files mentioned in this guide that not actually exist in the isp-imx package, like:
2.5.3 Sensor Driver API
V4l2 Sensor Driver API is declared in file <sensor>_mipi_v3.c , where <sensor> is the name of the sensor (for example, OV2775).
Basically all the topics I found in this forum recommends to create a copy of the existing and supported ov2775 files, adjust the code and modify the appropriate files. I've continued in this direction, using isp-imx-4.2.2.15.0 and yocto build system. I should notice that I'm new to yocto (but old buildroot user
Anyway, the list of the modified files and directories is:
./units/isi/drv/OV5640
./units/isi/drv/OV5640/source/OV5640.c
./units/isi/drv/OV5640/calib/OV5640
./units/isi/drv/OV5640/calib/OV5640/OV5640.xml
./units/isi/drv/Sensor1_Entry.cfg
./units/isi/drv/Sensor0_Entry.cfg
./units/isi/drv/OV5640/CMakeLists.txt
./vvcam/v4l2/sensor/ov5640
./vvcam/v4l2/sensor/ov5640/ov5640_regs_1080p.h
./mediacontrol/server/CMakeLists.txt
./appshell/shell_libs/CMakeLists.txt
./imx/run.sh
./imx/start_isp.sh
If I understand correctly this will produce the desired .drv file and calibration .xml (which will be dummy/incorrect), the lib files, etc. and will populate the rootfs with them. The calibration will be something to worry at later stage. I will change the appropriate dtb file to support the sensor on one of the available MIPI CSI ports later. Please, let me know if I'm missing something here?
Right now the isp-imx is compiled correctly. But at package_split stage of this package processing the yocto build is returning error related to the position of libov5640.so - it should be in dev, but isn't. So I'm stuck here.
The error output of bitbake is as follow:
ERROR: isp-imx-4.2.2.15.0-r0 do_package_qa: QA Issue: non -dev/-dbg/nativesdk- package isp-imx contains symlink .so '/usr/lib/libov5640.so' [dev-so]
ERROR: isp-imx-4.2.2.15.0-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.
ERROR: Logfile of failure stored in: /workdir/build_wayland/tmp/work/cortexa53-crypto-mx8mp-poky-linux/isp-imx/4.2.2.15.0-r0/temp/log.do_package_qa.18024
ERROR: Task (/workdir/sources/meta-optix/recipes-bsp/isp-imx/isp-imx_4.2.2.15.0.bb:do_package_qa) failed with exit code '1'
The libov****.so files at this point in /tmp/work/...../isp-imx/4.2.2.15.0-r0:
./packages-split/isp-imx/usr/lib/libov2775.so.1.0.0
./packages-split/isp-imx/usr/lib/libov5640.so.1.0.0
./packages-split/isp-imx/usr/lib/libov2775.so.1
./packages-split/isp-imx/usr/lib/libov5640.so.1
./packages-split/isp-imx/usr/lib/libov5640.so
./packages-split/isp-imx-dbg/usr/lib/.debug/libov2775.so.1.0.0
./packages-split/isp-imx-dbg/usr/lib/.debug/libov5640.so.1.0.0
./packages-split/isp-imx-dev/usr/lib/libov2775.so
I'm not sure is it the yocto inducing the problem or I haven't done something properly in the adding the sensor in isp-imx and this is the result?
Some help here, please?
It will be really helpful if someone has added new sensor to isp-imx and is willing to share some repo and/or steps how it is done - it will help all with the same problem to track down their mistakes. I'm quite sure I have done some fatal mistakes until now, which will be fun to debug
Hello @imsOPX , I would like to use the OV5640 with the ISP. Would it be possible for you to share all the files you mention regarding porting this camera for the ISP? Even if they are not fully working, I'd be very interested.
Best Regards,
Enguerrand de Ribaucourt from Savoir-faire Linux
Hi @ederibaucourt ,
I'm sorry but it is not possible to share the files. Even if they were only for training (before porting the real sensors) - they are still property of my employer. In fact the sensor was working quite well with the basic drivers I have written for the ISP. If you have some specific questions and if I may help you - I will try, just post me a PM.
Have a nice day!
At the end I have the sensor up and running using the IMX8M Plus ISP and the result is quite acceptable. The problems were mostly my errors in the kernel driver module. Well, there is some problem with the AEC feature of the ISP, but I will start another topic for this.
@imsOPX In your previous posts you said the gst-launch presented no errors but no images. How did you solve this issue? I have met the similar problem with you. Thanks sincerely.
Hi @imsOPX
It would be really helpful if you can provide some points to my issue.
Im trying to integrate IMX219 with IMX8MP.
Now I'm able to get the video nodes in the v4l2-ctl command
root@imx8mpevk:/opt/imx8-isp/bin# v4l2-ctl --list-devices
():
/dev/v4l-subdev0
/dev/v4l-subdev3
/dev/v4l-subdev4
():
/dev/v4l-subdev1
/dev/v4l-subdev5
/dev/v4l-subdev6
(csi0):
/dev/v4l-subdev2
FSL Capture Media Device (platform:mxc-md):
/dev/media0
VIV (platform:viv0):
/dev/video2
VIV (platform:viv1):
/dev/video3
vsi_v4l2dec (platform:vsi_v4l2dec):
/dev/video1
vsi_v4l2enc (platform:vsi_v4l2enc):
/dev/video0
viv_media (platform:vvcam-video):
/dev/media1
root@imx8mpevk:/opt/imx8-isp/bin#
But when i try to run the gstreamer command , getting the following error
ndsinkmx8mpevk:/opt/imx8-isp/bin# gst-launch-1.0 -v v4l2src device=/dev/video2 ! "video/x-raw,format=YUY2,width=1920,height=1080, framerate=30/1" ! queue ! imxvideoconvert_g2d ! waylandsink
[ 2473.477604] enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[ 2473.484168] <<< Command : VIDIOC_QUERYCAP
[ 2473.488464] enter (( imx219_ioc_qcap ))
[ 2473.885333] enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[ 2473.892340] <<< Command : VIDIOC_QUERYCAP
[ 2473.896640] enter (( imx219_ioc_qcap ))
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 2474.301297] enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[ 2474.304047] ... Return Value = 0 .....
[ 2474.310120] <<< Command : VIDIOC_QUERYCAP
[ 2474.310124] enter (( imx219_ioc_qcap ))
[ 2474.489256] enter imx219_priv_ioctl ((( (cmd = -2140645888))))
[ 2474.493931] ... Return Value = 0 .....
[ 2474.500042] <<< Command : VIDIOC_QUERYCAP
[ 2474.508151] enter (( imx219_ioc_qcap ))
loading [IMX219.drv]...
[ 2474.593452] enter imx219_priv_ioctl ((( (cmd = 289))))
[ 2474.598881] <<< Command : VVSENSORIOC_QUERY
[ 2474.603443] enter (( imx219_ioc_query_mode ))
ERROR : [MediaPipeline] NativeSensor open error!
ERROR : [V4l2Event] initialize MediaPipeline error!
[ 2479.485515] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 2479.645507] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 2479.809525] viv_post_event: unsubscribed event id =14 type=0x08002000
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fract9
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=9
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction9
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio9
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)9
/GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fractx
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, intx
/GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoco[ 2479.973514] viv_post_event: unsubscribed event id =14 type=0x08002000
nvert_g2d0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string9
[ 2480.149502] viv_post_event: unsubscribed event id =7 type=0x08002000
[ 2480.313526] viv_post_event: unsubscribed event id =6 type=0x08002000
[ 2480.477526] viv_post_event: unsubscribed event id =10 type=0x08002000
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate required memory.
Additional debug info:
../git/sys/v4l2/gstv4l2src.c(659): gst_v4l2src_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Buffer pool activation failed
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../git/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:06.475932375
Setting pipeline to NULL ...
[ 2480.653516] viv_post_event: unsubscribed event id =1 type=0x08002000
Freeing pipeline ...
root@imx8mpevk:/opt/imx8-isp/bin#
I think, we will get more debug info from isp-imx if we are able to enable the trace .
Can you please guide on this.
Best Regards
Charlie
actually I wasn't able to enable the TRACE globally (and I'm sure it will flood you with A LOT of information). So I did something simpler instead:
// ---> Start of CREATE_TRACER
enum{ IMX_VERBOSE = 0,
IMX_INFO,
IMX_WARN,
IMX_ERROR,
IMX_QUIET
};
#define IMX_DEBUG_LEVEL IMX_ERROR
#undef TRACE
#define TRACE(x, fmt, ...) \
do { \
if( x >= IMX_DEBUG_LEVEL ) \
printf(fmt, ##__VA_ARGS__); \
} while(0) \
// ---> End of CREATE_TRACER
....
TRACE( IMX_INFO, "%s: (enter)\n", __func__);
....
Well, it isn't very good solution, but it does the job - controlling the level of debug
@khang_letruongThanks for the support, I appreciate it. So far I've been able to execute start_isp.sh without errors, the log is as follows.
/opt/imx8-isp/bin# ./start_isp.sh
Starting isp_media_server with OV5640
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "ov5640"...
Removing vvcam-isp...
[ 142.597453] enter viv_isp_exit_module
[ 142.601271] enter isp_hw_remove
[ 142.604439] vvcam isp driver removed
Removing vvcam-dwe...
[ 142.641720] enter viv_dwe_exit_module
[ 142.645517] enter dwe_hw_remove
[ 142.648798] enter fake_pdev_release
[ 142.652319] vvcam dewarp driver removed
Removing vvcam-video...
Removing imx8-media-dev...
[ 142.706337] : Unregistered all entities
Removing ov5640...
[ 142.752475] enter ov5640_remove
[ 142.755626] ov5640 power off
[ 142.758829] enter ov5640_regulator_disable
[ 142.763409] enter ov5640_remove
[ 142.766595] ov5640 power off
[ 142.769964] enter ov5640_regulator_disable
Removing ov2775...
Removing os08a20...
Removing basler-camera-driver-vvcam...
Loading module ov5640 ...
ov5640
[ 142.923721] enter ov5640_probe
[ 142.927950] enter ov5640_retrieve_capture_properties
[ 142.932955] ov5640 1-003c: supply DOVDD not found, using dummy regulator
[ 142.939948] ov5640 1-003c: supply DVDD not found, using dummy regulator
[ 142.946767] ov5640 1-003c: supply AVDD not found, using dummy regulator
[ 142.953459] enter ov5640_set_clk_rate
[ 142.957161] ov5640 power on
[ 142.960239] ov5640 resetting
[ 143.048885] Chip ID = 0x5640
[ 143.051781] ov5640_probe camera mipi ov5640, is found
[ 143.057043] enter ov5640_probe
[ 143.061194] enter ov5640_retrieve_capture_properties
[ 143.066198] ov5640 3-003c: supply DOVDD not found, using dummy regulator
[ 143.073060] ov5640 3-003c: supply DVDD not found, using dummy regulator
[ 143.079780] ov5640 3-003c: supply AVDD not found, using dummy regulator
[ 143.086499] enter ov5640_set_clk_rate
[ 143.090197] ov5640 power on
[ 143.093276] ov5640 resetting
[ 143.184881] Chip ID = 0x5640
[ 143.187777] ov5640_probe camera mipi ov5640, is found
Loaded /lib/modules/5.10.52+gf455d15ccde8/extra/sensor/ov5640/ov5640.ko
Loading module imx8-media-dev ...
imx8_media_dev
[ 143.224037] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[ 143.234847] mx8-img-md: Registered sensor subdevice: ov5640 3-003c (1)
[ 143.241455] mx8-img-md: Registered sensor subdevice: ov5640 1-003c (2)
[ 143.248057] enter ov5640_link_setup
[ 143.251558] mx8-img-md: created link [ov5640 1-003c] => [mxc-mipi-csi2.0]
[ 143.258371] enter ov5640_link_setup
[ 143.261885] mx8-img-md: created link [ov5640 3-003c] => [mxc-mipi-csi2.1]
[ 143.268694] mxc-md 32c00000.bus:camera: mxc_md_create_links
[ 143.277847] ov5640_priv_ioctl: CMD = 0x80685600
[ 143.280641] ov5640_priv_ioctl: CMD = 0x80685600
[ 143.285231] enter ov5640_query_capability
Loaded /lib/modules/5.10.52+gf455[ 143.286962] enter ov5640_query_capability
d15ccde8/kernel/drivers/staging/media/imx/imx8-media-dev.ko
Loading module vvcam-video ...
vvcam_video
[ 143.329094] enter viv_video_init_module
Loaded /lib/modules/5.10.52+gf455d15ccde8/extra/video/vvcam-video.ko
Loading module vvcam-dwe ...
vvcam_dwe
[ 143.366613] enter viv_dwe_init_module
[ 143.370573] enter dwe_hw_probe
[ 143.373957] vvcam dewarp driver probed
Loaded /lib/modules/5.10.52+gf455d15ccde8/extra/vvcam-dwe.ko
Loading module vvcam-isp ...
vvcam_isp
[ 143.409671] enter viv_isp_init_module
[ 143.413712] enter isp_hw_probe
[ 143.417172] vvcam isp driver registered
Loaded /lib/modules/5.10.52+gf455d15ccde8/extra/vvcam-isp.ko
Starting isp_media_server with c[ 143.428030] enter isp_mi_stop
onfiguration file CAMERA0
When I try to play the video stream using gstreamer there are no errors. The log is:
/opt/imx8-isp/bin# gst-launch-1.0 -v v4l2src device=/dev/video0 ! "video/x-raw,format=YUY2,width=3840,height=2160" ! queue ! imxvideoconvert_g2d ! waylandsink
[ 363.172878] ov5640_priv_ioctl: CMD = 0x80685600
[ 363.177423] enter ov5640_query_capability
[ 363.181688] ov5640_priv_ioctl: CMD = 0x80685600
[ 363.186257] enter ov5640_query_capability
[ 363.200339] enter isp_mi_stop
Setting pipeline to PAUSED ...
[ 363.269814] ov5640_priv_ioctl: CMD = 0x80685600
Pipeline is live and does not nee[ 363.274821] enter ov5640_query_capability
d PREROLL ...
Pipeline is PREROL[ 363.281727] ov5640_priv_ioctl: CMD = 0x80685600
LED ...
Setting pipeline to PLAY[ 363.289094] enter ov5640_query_capability
ING ...
New clock: GstSystemClock
[ 363.340146] ov5640_priv_ioctl: CMD = 0x80685600
[ 363.344703] enter ov5640_query_capability
[ 363.348798] ov5640_priv_ioctl: CMD = 0x80685600
[ 363.353375] enter ov5640_query_capability
loading [ov5640.drv]...
OV5640_IsiHalQuerySensorIss (en[ 363.369883] ov5640_priv_ioctl: CMD = 0x105
ter)
[ 363.376855] enter ov5640_query_supports
OV5640_IsiHalQuerySensorIss (exit[ 363.381900] ov5640_priv_ioctl: CMD = 0x80685600
)
WARN : [DEWARP_HW][ 363.388755] enter ov5640_query_capability
dwe type /dev/v4l-subdev0 0 [ 363.395665] ov5640_priv_ioctl: CMD = 0x80685600
WARN : [DEWARP_HW] [ 363.403047] enter ov5640_query_capability
dwe type /dev/v4l-subdev1 1
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev2 2
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev3 3
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev4 4
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mo1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interl1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=[ 363.527092] ov5640_priv_ioctl: CMD = 0x105
(int)3840, height=(int)2160, fram[ 363.532622] enter ov5640_query_supports
erate=(fraction)30/1, pixel-aspec[ 363.539386] ov5640_priv_ioctl: CMD = 0x101
t-ratio=(fraction)1/1, interlace-[ 363.546333] ov5640_priv_ioctl: CMD = 0x104
mode=(string)progressive, colorim[ 363.553299] ov5640_priv_ioctl: CMD = 0x103
etry=(string)2:6:11:1
/GstPipeli[ 363.560261] ov5640_priv_ioctl: CMD = 0x100
ne:pipeline0/imxvideoconvert_g2d:[ 363.567218] ov5640_priv_ioctl: CMD = 0x106
imxvideoconvert_g2d0.GstPad:src: [ 363.574205] enter ov5640_set_sensor_mode
caps = video/x-raw, width=(int)38[ 363.580987] ov5640_priv_ioctl: CMD = 0x107
40, height=(int)2160, framerate=([ 363.587950] enter ov5640_get_sensor_mode
fraction)30/1, pixel-aspect-ratio[ 363.594785] ov5640_priv_ioctl: CMD = 0x105
=(fraction)1/1, interlace-mode=(s[ 363.601720] enter ov5640_query_supports
tring)progressive, colorimetry=(s[ 363.609673] enter isp_mi_stop
tring)2:6:11:1, format=(string)BG[ 363.616828] enter ov5640_set_fmt
Rx
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = video/x-raw, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)px
/GstPipeline:pipeline0/imxvideoconvert_g2d:imxvideoconvert_g2d0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, [ 363.652933] ov5640_set_fmt: 1920x1080
height=(int)2160, framerate=(frac[ 363.659203] enter ov5640_get_format_code
tion)30/1, pixel-aspect-ratio=(fr[ 363.665988] ov5640_get_format_code: pattern = 0x3 | bit_width = 0xa
action)1/1, interlace-mode=(strin[ 363.675157] ov5640_priv_ioctl: CMD = 0x11d
g)progressive, colorimetry=(strin[ 363.682066] enter ov5640_set_test_pattern
g)2:6:11:1
isp input: 1920x1080[ 363.689249] ov5640_priv_ioctl: CMD = 0x117
RAW10
isp output: 1920x1080 YUV[ 363.695864] enter ov5640_set_fps
422I
dewarp input: 1920x1080 Y[ 363.702518] ov5640_priv_ioctl: CMD = 0x107
UV422I
dewarp output: 3840x2160[ 363.708915] enter ov5640_get_sensor_mode
YUV422I
OV5640_IsiGetSensorIss [ 363.716046] ov5640_priv_ioctl: CMD = 0x112
(enter)
OV5640_IsiGetSensorIss ([ 363.722635] enter ov5640_set_exp
exit)
OV5640_IsiHalQuerySensorIs[ 363.730392] ov5640_priv_ioctl: CMD = 0x115
s (enter)
OV5640_IsiHalQuerySen[ 363.735669] enter ov5640_set_gain
sorIss (exit)
OV5640_IsiCreateSe[ 363.743553] enter isp_s_comp
nsorIss (enter)
OV5640_IsiSensor[ 363.747610] enter isp_s_comp
SetPowerIss: (enter)
OV5640_IsiS[ 363.753350] enter isp_s_comp
ensorSetPowerIss: set power 1
OV5640_IsiSensorSetPowerIss: (exit)
OV5640_IsiSensorGetClkIss: (enter)
OV5640_IsiSensorGetClkIss: status:276050132 sensor_mc[ 363.772367] enter wdr3_hw_init
lk:24000000 csi_max_pixel_clk:500[ 363.775730] wdr3 res: 1920 1080
000000
OV5640_IsiSensorGetClkIss[ 363.782417] enter isp_mi_start
: (exit)
OV5640_IsiSensorSetClkI[ 363.788205] ov5640_priv_ioctl: CMD = 0x110
ss: (enter)
OV5640_IsiSensorSetC[ 363.794727] ov5640_s_stream: 1
lkIss: status:1 sensor_mclk:24000000 csi_max_pixel_clk:500000000
OV5640_IsiSensorSetClkIss: (exit)
OV5640_IsiResetSensorIss: (enter)
OV5640_IsiResetSensorIss: (exit)
OV5640_IsiSetSensorModeIss (enter)
OV5640_IsiSetSensorModeIss (exit)
OV5640_IsiCreateSensorIss (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit)
OV5640_IsiGetCapsIss (enter)
OV5640_IsiQuerySensorIss (enter)
OV5640_IsiHalQuerySensorIss (enter)
OV5640_IsiHalQuerySensorIss (exit)
OV5640_IsiQuerySensorIss (exit)
OV5640_IsiGetCapsIss (exit)
OV5640_IsiSetupSensorIss (enter)
OV5640_IsiSetupSensorIss (exit)
OV5640_IsiSetTestPatternIss (enter)
OV5640_IsiSetTestPatternIss: test pattern enable[0] mode[0]
OV5640_IsiSetTestPatternIss: (exit)
OV5640_IsiSetSensorFpsIss: (enter)
OV5640_IsiSetSensorFpsIss: (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit)
OV5640_IsiGetAeInfoIss (enter)
OV5640_IsiGetAeInfoIss (exit)
OV5640_IsiGetSensorFpsIss: (enter)
OV5640_IsiGetSensorFpsIss: (exit)
OV5640_IsiGetAeStartExposureIs (enter)
OV5640_IsiGetAeStartExposureIs:get start exposure 36403200
OV5640_IsiGetAeStartExposureIs: (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit)
OV5640_IsiGetAeInfoIss (enter)
OV5640_IsiGetAeInfoIss (exit)
OV5640_IsiSetIntegrationTimeIss (enter)
OV5640_IsiSetIntegrationTimeIss set linear exp 400
OV5640_IsiSetIntegrationTimeIss (exit)
OV5640_IsiSetGainIss (enter)
OV5640_IsiSetGainIss set linear gain 3072
OV5640_IsiSetGainIss (exit)
OV5640_IsiGetIntegrationTimeIss (enter)
OV5640_IsiGetIntegrationTimeIss (exit)
OV5640_IsiGetGainIss (enter)
OV5640_IsiGetGainIss (exit)
#### CsiSetFormat 1920 1080 808535874
OV5640_IsiGetSensorIspStatusIss: (enter)
OV5640_IsiGetSensorIspStatusIss: (exit)
OV5640_IsiGetSensorIspStatusIss: (enter)
OV5640_IsiGetSensorIspStatusIss: (exit)
OV5640_IsiGetSensorIspStatusIss: (enter)
OV5640_IsiGetSensorIspStatusIss: (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit)
OV5640_IsiGetSensorIspStatusIss: (enter)
OV5640_IsiGetSensorIspStatusIss: (exit)
CamEngineEnableCproc: set val 1 1 1 1.10 -15 1.00 0.00
OV5640_IsiSensorSetStreamingIss (enter)
OV5640_IsiSensorSetStreamingIss: set streaming 1
OV5640_IsiSensorSetStreamingIss (exit)
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:08.579723250
Setting pipeline to NULL ...
[ 371.891324] enter isp_mi_stop
OV5640_IsiSensorSetStreamingIss ([ 371.997447] ov5640_priv_ioctl: CMD = 0x110
enter)
[ 372.004419] ov5640_s_stream: 0
OV5640_IsiSensorSetStreamingIss: set streaming 0
OV5640_IsiSensorSetStreamingIss (exit)
OV5640_IsiGetSensorModeIss (enter)
OV5640_IsiGetSensorModeIss (exit)
OV5640_IsiReleaseSensorIss (enter[ 372.024207] ov5640_priv_ioctl: CMD = 0x110
)
OV5640_IsiSensorSetStreamingI[ 372.031192] ov5640_s_stream: 0
ss (enter)
[ 372.037709] ov5640_priv_ioctl: CMD = 0x104
OV5640_IsiSensorSetStreamingIss: [ 372.042286] ov5640_priv_ioctl: CMD = 0x103
set streaming 0
OV5640_IsiSensor[ 372.049251] ov5640_priv_ioctl: CMD = 0x101
SetStreamingIss (exit)
OV5640_IsiSensorGetClkIss: (enter)
OV5[ 372.060986] enter isp_mi_stop
640_IsiSensorGetClkIss: status:276050132 sensor_mclk:24000000 csi_max_pixel_clk:500000000
OV5640_IsiSensorGetClkIss: (exit)
OV5640_IsiSensorSetClkIss: (enter)
OV5640_IsiSensorSetClkIss: status:0 sensor_mclk:24000000 csi_max_pixel_clk:500000000
OV5640_IsiSensorSetClkIss: (exit)
OV5640_IsiSensorSetPowerIss: (enter)
OV5640_IsiSensorSetPowerIss: set power 0
OV5640_IsiSensorSetPowerIss: (exit)
OV5640_IsiReleaseSensorIss (exit)
Freeing pipeline ...
The logs contains some debug prints from my driver as you see, just to be sure the proper functions are called.
Well, so far there is no image.. and I'm thinking what can be wrong. Most probably my ov5640 driver isn't ok, at least to say. I'm using some values from ov5640_mipi.c linux kenel driver init structures, anyway - there are a lot of registers and it is easy to make mistake
Hi @imsOPX ,
Please share the output of following commands :
$ v4l2-ctl --list-devices
$ v4l2-ctl --device /dev/video0 --all
Regards,
Khang
@khang_letruong, here you are:
v4l2-ctl --list-devices
[ 2781.351229] ov5640_priv_ioctl: CMD = 0x80685600
[ 2781.355856] enter ov5640_query_capability
[ 2781.360079] ov5640_priv_ioctl: CMD = 0x80685600
[ 2781.364626] enter ov5640_query_capability
():
/dev/v4l-subdev0
/dev/v4l-subdev4
/dev/v4l-subdev5
():
/dev/v4l-subdev1
(csi0):
/dev/v4l-subdev3
(csi1):
/dev/v4l-subdev2
FSL Capture Media Device (platform:mxc-md):
/dev/media0
VIV (platform:viv0):
/dev/video0
viv_media (platform:vvcam-video):
/dev/media1
v4l2-ctl --device /dev/video0 --all
Driver Info:
Driver name : viv_v4l2_device
Card type : VIV
Bus info : platform:viv0
Driver version : 5.10.52
Capabilities : 0x84200001
Video Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Media Driver Info:
Driver name : vvcam-video
Model : viv_media
Serial :
Bus info :
Media version : 5.10.52
Hardware revision: 0x00000000 (0)
Driver version : 5.10.52
Interface Info:
ID : 0x03000003
Type : V4L Video
Entity Info:
ID : 0x00000001 (1)
Name : viv_v4l20
Function : V4L2 I/O
Pad 0x01000002 : 0: Sink, Must Connect
Link 0x02000010: from remote pad 0x1000006 of entity 'vvcam-dwe.0': Data, Enabled
Priority: 2
Video input : 0 (camera: ok)
Format Video Capture:
Width/Height : 3840/2160
Pixel Format : 'YUYV' (YUYV 4:2:2)
Field : None
Bytes per Line : 7680
Size Image : 16588800
Colorspace : Rec. 709
Transfer Function : Rec. 709
YCbCr/HSV Encoding: BT.2020
Quantization : Limited Range
Flags :
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 3840, Height 2160
Default : Left 0, Top 0, Width 3840, Height 2160
Pixel Aspect: 1/1
Selection Video Capture: crop, Left 0, Top 0, Width 3840, Height 2160, Flags:
Selection Video Capture: crop_default, Left 0, Top 0, Width 3840, Height 2160, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 3840, Height 2160, Flags:
Selection Video Capture: compose, Left 0, Top 0, Width 3840, Height 2160, Flags:
Selection Video Capture: compose_default, Left 0, Top 0, Width 3840, Height 2160, Flags:
Selection Video Capture: compose_bounds, Left 0, Top 0, Width 3840, Height 2160, Flags:
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: 30.000 (30/1)
Read buffers : 0
User Controls
viv_ext_ctrl 0x0098f901 (str) : min=0 max=65535 step=1 value='{\n\x09"dwe" : \n\x09{\n\x09\x09"bypass" : false,\n\x09\x09"hflip" : false,\n\x09\x09"mat" : \n\x09\x09[\n\x09\x09\x096d
WARN : [V4l2Event] extctrl2: cid=0x98f903,ret=0
sensor_resw 0x0098f903 (int) : min=0 max=3840 step=1 default=1920 value=1920 flags=read-only, volatile
WARN : [V4l2Event] extctrl2: cid=0x98f904,ret=0
sensor_resh 0x0098f904 (int) : min=0 max=2160 step=1 default=1080 value=1080 flags=read-only, volatile
Now I see there is something wrong with the max height and width - this sensor is capable of 2592x1944, not 3840x2160, anyway I have tried with 1920x1080, it is still the same result - no error, no image.
As I understand you have ported imx327 to isp-imx, right?
Hi @imsOPX ,
Yes, I did port imx296, imx327, ar0234 and other chinese sensors' drivers with 1440x1080 and 1920x1080 resolutions. Currently, I am porting a 4K one.
Regards,
Khang
Well, as expected, the concrete problem was aftermath of my mistake in the recipe for isp-imx. Always check your FILES_${PN}-dev += ...
Anyway, as this is only part of the process, I will like to left the topic open for further questions related to the porting process.
Hi @imsOPX ,
I am not sure if you have resolved your issue but there's example of how to integrate custom isp-imx into Yocto here : https://github.com/boundarydevices/meta-boundary/tree/master/recipes-bsp/isp-imx
Also, as for your issue I confirm the solution by adding following lines into isp-imx_%.bbappend as below :
FILES_${PN}-dev += " \
${libdir}/lib<custom-sensor-name>.so \
"
Regards,
Khang
Thank you for your answer, @khang_letruong
Yes, this was the problem, as I mentioned above.
Now I'm trying to understand why it is not working at all. Right now I'm attempting to enable trace prints in my driver as part of the ISP. I already enabled with 1s TRACE( .., .., 1) prints, but nothing shows up/ I'm a bit of confused is the ISP framework calling the correct functions at all? I can see the kernel driver for my sensor up, but the "glue" .drv functions are strangely quiet.
I have disabled the imx8-isp.service and I'm starting it manually with start_isp.sh:
root@imx8mp-var-dart:/opt/imx8-isp/bin# ./start_isp.sh
Starting isp_media_server with OV5640 Optix
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "optix_ov5640"...
Removing vvcam-isp...
Removing vvcam-dwe...
Removing vvcam-video...
[ 917.253749] enter viv_video_exit_module
Removing imx8-media-dev...
[ 917.396216] : Unregistered all entities
Removing ov5640...
Removing ov2775...
Removing os08a20...
Removing basler-camera-driver-vvcam...
Loading module ov5640 ...
ov5640
[ 917.538486] ov5640 1-003c: supply DOVDD not found, using dummy regulator
[ 917.545763] ov5640 1-003c: supply AVDD not found, using dummy regulator
[ 917.552521] ov5640 1-003c: supply DVDD not found, using dummy regulator
[ 917.601445] ov5640 3-003c: supply DOVDD not found, using dummy regulator
[ 917.608443] ov5640 3-003c: supply AVDD not found, using dummy regulator
[ 917.615299] ov5640 3-003c: supply DVDD not found, using dummy regulator
Loaded /lib/modules/5.10.52+ga18cf3faaeed/kernel/drivers/media/i2c/ov5640.ko
Loading module imx8-media-dev ...
imx8_media_dev
[ 917.701182] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[ 917.712059] mx8-img-md: Registered sensor subdevice: ov5640 3-003c (1)
[ 917.718649] mx8-img-md: Registered sensor subdevice: ov5640 1-003c (2)
[ 917.725211] mx8-img-md: created link [ov5640 1-003c] => [mxc-mipi-csi2.0]
[ 917.732014] mx8-img-md: created link [ov5640 3-003c] => [mxc-mipi-csi2.1]
[ 917.738844] mxc-md 32c00000.bus:camera: mxc_md_create_links
Loaded /lib/modules/5.10.52+ga18cf3faaeed/kernel/drivers/staging/media/imx/imx8-media-dev.ko
Loading module vvcam-video ...
vvcam_video
[ 917.777826] enter viv_video_init_module
Loaded /lib/modules/5.10.52+ga18cf3faaeed/extra/video/vvcam-video.ko
Loading module vvcam-dwe ...
vvcam_dwe
vvcam_dwe 28672 2[ 917.799077] enter isp_mi_stop
vvcam-dwe already loaded.
Loading module vvcam-isp ...
vvcam_isp
vvcam_isp 69632 2
vvcam-isp already loaded.
Starting isp_media_server with configuration file CAMERA0
INFO : [ISP_MEDIA_SERVER] ******************************************************************
INFO : [ISP_MEDIA_SERVER] VIV ISP Media Control Framework V4.2.2p15 (Aug 1 2022, 13:09:03)
INFO : [ISP_MEDIA_SERVER] ******************************************************************
And the result of v4l2-ctl --all
root@imx8mp-var-dart:/opt/imx8-isp/bin# v4l2-ctl --all
INFO : [MediaPipeline] NativeSensor Creat
[ 1015.143134] enter isp_mi_stop
ERROR : [MediaPipeline] NativeSensor open error!
ERROR : [V4l2Event] initialize MediaPipeline error!
Driver Info:
Driver name : viv_v4l2_device
Card type : VIV
Bus info : platform:viv0
Driver version : 5.10.52
Capabilities : 0x84200001
Video Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Media Driver Info:
Driver name : vvcam-video
Model : viv_media
Serial :
Bus info :
Media version : 5.10.52
Hardware revision: 0x00000000 (0)
Driver version : 5.10.52
Interface Info:
ID : 0x03000003
Type : V4L Video
Entity Info:
ID : 0x00000001 (1)
Name : viv_v4l20
Function : V4L2 I/O
Pad 0x01000002 : 0: Sink, Must Connect
Link 0x0200000a: from remote pad 0x1000009 of entity 'vvcam-isp.0': Data, Enabled
Priority: 2
Video input : 0 (camera: ok)
Format Video Capture:
Width/Height : 0/0
Pixel Format : ''
Field : None
Bytes per Line : 0
Size Image : 0
Colorspace : Rec. 709
Transfer Function : Default (maps to Rec. 709)
YCbCr/HSV Encoding: Default (maps to Rec. 709)
Quantization : Default (maps to Full Range)
Flags :
[ 1020.286862] viv_post_event: unsubscribed event id =14 type=0x08002000
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 0, Height 0
Default : Left 0, Top 0, Width 0, Height 0
Pixel Aspect: 1/1
[ 1020.446856] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: crop, Left 0, Top 0, Width 0, Height 0, Flags:
[ 1020.610862] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: crop_default, Left 0, Top 0, Width 0, Height 0, Flags:
[ 1020.774862] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 0, Height 0, Flags:
[ 1020.938869] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: compose, Left 0, Top 0, Width 0, Height 0, Flags:
[ 1021.102863] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: compose_default, Left 0, Top 0, Width 3840, Height 2160, Flags:
[ 1021.266863] viv_post_event: unsubscribed event id =14 type=0x08002000
Selection Video Capture: compose_bounds, Left 0, Top 0, Width 3840, Height 2160, Flags:
[ 1021.430862] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 1021.594862] viv_post_event: unsubscribed event id =14 type=0x08002000
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: invalid (0/1)
Read buffers : 0
User Controls
viv_ext_ctrl 0x0098f901 (str) : min=0 max=65535 step=1 value='' flags=has-payload
[ 1021.758862] viv_post_event: unsubscribed event id =14 type=0x08002000
error 22 getting ctrl sensor.resw
[ 1021.922863] viv_gen_g_ctrl: unsubscribed event id =11 type=0x08002000
error 22 getting ctrl sensor.resh
[ 1022.086863] viv_gen_g_ctrl: unsubscribed event id =11 type=0x08002000
And I'm stuck here..
Hi imsOPX
I have two questions.
First is Why did you port Ov5640 to the isp-imx ? because the driver of the ov5649 is already ported in the kernel source below
drivers/media/platform/mxc/capture/ov5640_mipi_v2.c
drivers/media/platform/mxc/capture/ov5640_v2.c
drivers/media/i2c/ov5640.c
Did you have any reason to port the ov5640 sensor to the isp-imx package ??
Second, I have a custom board whose cpu is imx8mp, there is a ISP and the imx273 image sensor.
This ISP controls the imx273 sensor thru I2C and i assume i can get image stream thru MIPI and V4L2 buffer.
This ISP provides several functionalities and I2C interface to communicate.
As a result, the control path is different with that of the isp-imx. I wonder I have to port this ISP to the isp-imx package of the NXP ?
Do you think There is a need i have to port the ISP to the isp-imx ??
I want to hear of your idea. i will wait your reply. previously thanks
Hi @imsOPX
I was trying to port the IMX219 with the IMX8MP. Can you please guide me to enable the traceing in the isp-imx?
Regards
Charlie
Hi again,
For following error:
root@imx8mp-var-dart:/opt/imx8-isp/bin# v4l2-ctl --all
INFO : [MediaPipeline] NativeSensor Creat
[ 1015.143134] enter isp_mi_stop
ERROR : [MediaPipeline] NativeSensor open error!
ERROR : [V4l2Event] initialize MediaPipeline error!
Maybe you should compare the driver name included in your .drv file (or its source <sensor-name>.c such as OV5640.c) and the driver name in the XML file(s) as well.
Regards,
K.
Hi @imsOPX,
I already enabled with 1s TRACE( .., .., 1) prints, but nothing shows up/ I'm a bit of confused is the ISP framework calling the correct functions at all
Should you also need to export the ISP_LOG_LEVEL=8?
According to my experience, in older version (isp-imx-4.2.2.11.0), it was not straight-forward to enable to tracing by just setting 1s to TRACE() macros. I am not sure if they (NXP) have been more "open" on that in recent versions. Also they confirmed to me that the Full build had not been released to customers : https://community.nxp.com/t5/i-MX-Processors/IMX8MP-isp-imx-Undefined-symbol-trace/m-p/1312440#M1773...
I was able to enable to tracing of isp-imx via their professional support channel, however.
Regards,
K.