topic Integrating Sony IMX586 Camera on IMX8MP platform in i.MX Processors

Integrating Sony IMX586 Camera on IMX8MP platform

BillR1 — Tue, 07 May 2024 20:37:35 GMT

Hi, I have followed the standard porting proceedure for the IMX586 sensor. It seems like for the most part "all systems go" except for the error at the end when the the adapter-0 cannot be opened.

Start the ISP media server
-----------------------------------------------------------------------

root@imx8mp:~# /opt/imx8-isp/bin/start_isp.sh
Starting isp_media_server for IMX586
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "imx586_4k_60"...
Removing vvcam-video...
Removing vvcam-isp...
Removing vvcam-dwe...
Removing imx8-media-dev...
Removing ov2775...
Removing os08a20...
Removing imx586 basler-camera-driver-vvcam...
Loading module imx586 ...
imx586
Loaded /lib/modules/6.1.36+/extra/imx586.ko
Loading module imx8-media-dev ...
imx8_media_dev
Loaded /lib/modules/6.1.36+/kernel/drivers/staging/media/imx/imx8-media-dev.ko
Loading module vvcam-dwe ...
vvcam_dwe
Loaded /lib/modules/6.1.36+/extra/vvcam-dwe.ko
Loading module vvcam-isp ...
vvcam_isp
Loaded /lib/modules/6.1.36+/extra/vvcam-isp.ko
Loading module vvcam-video ...
vvcam_video
Loaded /lib/modules/6.1.36+/extra/vvcam-video.ko
Starting isp_media_server with configuration file CAMERA0

...

Kernel output during the above
---------------------------------------------------------------------

[ 142.618951] enter isp_hw_remove
[ 142.622407] vvcam isp driver removed
[ 142.626234] enter isp_hw_remove
[ 142.629903] vvcam isp driver removed
[ 142.678850] enter viv_dwe_exit_module
[ 142.682740] enter dwe_hw_remove
[ 142.686024] enter fake_pdev_release
[ 142.689957] vvcam dewarp driver removed
[ 142.742167] : Unregistered all entities
[ 142.818522] Enter imx586_remove
[ 142.821677] imx586_power_off is bypassed, power fixed on for now
[ 142.827719] imx586 4-001a: imx586_regulator_disable todo
[ 142.966247] Enter imx586_probe
[ 142.969347] Setting mclk to 25 MHz
[ 142.974656] camera sensor chip id reported as 0x586
[ 142.979584] media_entity_pads_init()
[ 142.983180] v4l2_async_register_subdev_sensor()
[ 143.009429] imx586_probe camera mipi imx586, is found
[ 143.053308] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[ 143.064125] mx8-img-md: Registered sensor subdevice: imx586 4-001a (1)
[ 143.070692] mx8-img-md: created link [imx586 4-001a] => [mxc-mipi-csi2.0]
[ 143.077497] mxc-md 32c00000.bus:camera: mxc_md_create_links
[ 143.096969] Enter imx586_priv_ioctl, command 2154321408 (0x80685600)
[ 143.097687] Entering mipi_csis_imx8mp_phy_reset
[ 143.103634] Enter imx586_query_capability
[ 143.112371] imx586_query_capability, adapter 30ad0000.i2c (4) is found
[ 143.126133] enter viv_dwe_init_module
[ 143.130124] enter dwe_hw_probe
[ 143.133621] vvcam dewarp driver probed
[ 143.177634] enter viv_isp_init_module
[ 143.181614] enter isp_hw_probe
[ 143.185075] vvcam isp driver registered
[ 143.189129] enter isp_hw_probe
[ 143.192676] vvcam isp driver registered
[ 143.242058] enter viv_video_init_module
[ 143.282512] enter isp_mi_stop
[ 143.286685] enter isp_mi_stop

root@imx8mp:~# v4l2-ctl --list-devices
():
/dev/v4l-subdev0
/dev/v4l-subdev2
/dev/v4l-subdev3

():
/dev/v4l-subdev4
/dev/v4l-subdev5

(csi0):
/dev/v4l-subdev1

FSL Capture Media Device (platform:32c00000.bus:camera):
/dev/media0

VIV (platform:viv0):
/dev/video2

VIV (platform:viv1):
/dev/video3

vsi_v4l2dec (platform:vsi_v4l2dec):
/dev/video1

vsi_v4l2enc (platform:vsi_v4l2enc):
/dev/video0

viv_media (platform:vvcam-video.0):
/dev/media1

Then try something

root@imx8mp:~# v4l2-ctl --device=/dev/video2 --all

Output from media server:

ADAPTER: AdaptOpen:open sensor failed in v4l2 mode! check sensor driver, cfg is imx586-vvcam
HAL-ALTERA: HalOpen: open adapter-0 failed in HAL
[CITF][ERR]: CitfApi create error!
[CITF][ERR]: initHardware new CitfApi 0 fail!
ERROR : [NativeSensor] mDriver.initHardware failed, id 0
ERROR : [MediaPipeline] NativeSensor open error!
ERROR : [V4l2Event] initialize MediaPipeline error!

I assume the following is incomplete because it isn't calling my driver properly.

root@imx8mp:~# v4l2-ctl --device=/dev/video2 --all
Driver Info:
Driver name : viv_v4l2_device
Card type : VIV
Bus info : platform:viv0
Driver version : 6.1.36
Capabilities : 0x84201001
Video Capture
Video Capture Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Priority: 2
Video input : 0 (camera: ok)
Format Video Capture:
Width/Height : 0/0
Pixel Format : ''
Field : None
Bytes per Line : 0
Size Image : 0
Colorspace : SMPTE 170M
Transfer Function : Default (maps to Rec. 709)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Full Range)
Flags :
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 0, Height 0
Default : Left 0, Top 0, Width 0, Height 0
Pixel Aspect: 1/1
Selection Video Capture: crop, Left 0, Top 0, Width 0, Height 0, Flags:
Selection Video Capture: crop_default, Left 0, Top 0, Width 0, Height 0, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 0, Height 0, Flags:
Selection Video Capture: compose, Left 0, Top 0, Width 0, Height 0, Flags:
Selection Video Capture: compose_default, Left 0, Top 0, Width 0, Height 0, Flags:
Selection Video Capture: compose_bounds, Left 0, Top 0, Width 0, Height 0, Flags:
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: invalid (0/0)
Read buffers : 0

User Controls

viv_ext_ctrl 0x0098f901 (str) : min=0 max=65535 step=1 value='' flags=has-payload

The driver output during the above query shows the following.

[ 966.535532] Enter imx586_priv_ioctl, command 0x80685600
[ 966.546583] Enter imx586_query_capability
[ 966.550606] imx586_query_capability, adapter 30ad0000.i2c (4) is found
[ 966.565855] enter isp_mi_stop
[ 966.645776] enter isp_mi_stop

Seems like the only private ioctl that gets called in my driver is VIDIOC_QUERYCAP. Seems like this should be something basic I am missing. I noticed mention in HAL-ALTERA of adapter-0 and my adapter is reported as 4, if they are the same thing.

Bill

Re: Integrating Sony IMX586 Camera on IMX8MP platform

joanxie — Wed, 08 May 2024 01:58:44 GMT

refer to your logfile, open sensor failed in v4l2 mode! check sensor driver,it seems this error is related to your camera driver you need check your camera driver firstly

Re: Integrating Sony IMX586 Camera on IMX8MP platform

BillR1 — Wed, 08 May 2024 13:37:26 GMT

Well I did check the driver originally first. Then I decided to print out the v4l2 capabilities and got the following so I think I am on to something here.

[63477.135304] Enter imx586_query_capability
[63477.143351] Driver: imx586
[63477.146058] Card:
[63477.148082] Bus Info: csi0
[63477.150800] Version: 0
[63477.153160] Capabilities: 0
[63477.155961] Device Caps: 0
[63477.158678] imx586_query_capability, adapter 30ad0000.i2c (4) is found

Re: Integrating Sony IMX586 Camera on IMX8MP platform

BillR1 — Wed, 08 May 2024 14:52:51 GMT

Unfortunately missing capabilities was not the problem. (Also the other drivers do not do it either). Also it appears that the error message is coming from the proprietary libcam_engine.so driver so I can not look to the source to find out what its not happy about. This is the point just before it should load imx586.drv.

[65062.927912] Entering mipi_csis_imx8mp_phy_reset
[65062.928038] Enter imx586_priv_ioctl, command -2140645888 #query caps
[65062.938077] imx586_query_capability, adapter 30ad0000.i2c (4) is found
[65062.944634] Driver: imx586
[65062.947379] Card: IMX586 Card
[65062.950369] Bus Info: csi0
[65062.953169] Version: 65536
[65062.955904] Capabilities: 67108865
[65062.959345] Device Caps: 67108865
[65063.037301] enter isp_mi_stop
ADAPTER: AdaptOpen:open sensor failed in v4l2 mode! check sensor driver, cfg is imx586-vvcam
HAL-ALTERA: HalOpen: open adapter-0 failed in HAL
[CITF][ERR]: CitfApi create error!
[CITF][ERR]: initHardware new CitfApi 0 fail!
ERROR : [NativeSensor] mDriver.initHardware failed, id 0
ERROR : [MediaPipeline] NativeSensor open error!
ERROR : [V4l2Event] initialize MediaPipeline error!

Re: Integrating Sony IMX586 Camera on IMX8MP platform

joanxie — Sat, 11 May 2024 05:41:14 GMT

you should focus on the v4l2 mode in the sensor driver of ISP, you can refer to the chapter 6.2 Camera Sensor Porting Setup in V4L2 Mode of enclosed file, you also can refer to the IMX219 package as reference

https://github.com/voxelbotics/imx8mp-imx219-sensor-enablement

Re: Integrating Sony IMX586 Camera on IMX8MP platform

BillR1 — Sat, 11 May 2024 17:07:58 GMT

I made a regular v4l2 style driver on ISI. Its revealed some issues but now its not just black box of the isp_media_server proprietary libs telling me to check my driver. I have gotten a little further with it for now.

Bill

Re: Integrating Sony IMX586 Camera on IMX8MP platform

AndrejsVasiljevs — Tue, 25 Mar 2025 23:40:17 GMT

Hi Bill! @bily

Just wondering have you managed the error you've mentioned here?

I've faced the same error message and have no ideas where to dig...

Any hint would be highly appreciated.

Re: Integrating Sony IMX586 Camera on IMX8MP platform

BillR1 — Wed, 26 Mar 2025 12:47:15 GMT

Hi, if I understand your issue correctly:

ADAPTER: AdaptOpen:open sensor failed in v4l2 mode! check sensor driver, cfg is imx586-vvcam HAL-ALTERA: HalOpen: open adapter-0 failed in HAL [CITF][ERR]: CitfApi create error! [CITF][ERR]: initHardware new CitfApi 0 fail! ERROR : [NativeSensor] mDriver.initHardware failed, id 0 ERROR : [MediaPipeline] NativeSensor open error! ERROR : [V4l2Event] initialize MediaPipeline error!

my problem was that the camera sensor name in Sensor0_Entry.cfg didn't match the vvcam driver, in my case imx586_mipi_v3.c the imx586_query_capability() function.

Regards, Bill

Re: Integrating Sony IMX586 Camera on IMX8MP platform

AndrejsVasiljevs — Wed, 26 Mar 2025 13:14:59 GMT

Thanks a lot for your reply.

Just for historical purposes my error was:

ADAPTER: AdaptOpen:open sensor failed in v4l2 mode! check sensor driver, cfg is os08a20 HAL-ALTERA: HalOpen: open adapter-1 failed in HAL [CITF][ERR]: CitfApi create error! [CITF][ERR]: initHardware new CitfApi 1 fail!

The main difference from basic scenarios is that it is dual camera scenario where CAM0 connected via ISI and CAM1 via ISP. Linux version 6.1 with imx8-isp from version 6.6 (version 6.1 has a bug in such scenarious)

The issue was solved by reordering driver modules load sequence.
Upon restart some of modules automatically loaded and thus I've removed them from imx8-isp start script. But, actually, everything works when all related modules (imx8-media-dev, sensors modules, vvcam-*) unloaded and reloaded again from imx8-isp start script.

IMHO, something wrong in kernel driver logic with modules linkage. So far I've left it as it is.