topic Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform in i.MX Processors

Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Thu, 17 Oct 2024 06:54:08 GMT

HI,

We have supported UYVY, SRGB10 formats from the camera and using iMX8MPlus kit for the evaluation.

After booting of the kit, if we first launch either UYVY or SRGB10, the stream works without any issues in the format. But when we try to switch between the format, the stream stops.

We have debugged the camera MIPI signals. It is correctly configuring with the required format and MIPI signal activity is proper in the camera.

But the iMX8Mplus platform not supporting the switching between the different formats. Is there any patches to resolve this issue?

We are using following kernel.
kernel : v6.1.55
Yocto : mickledore

Thanks.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Fri, 18 Oct 2024 11:21:12 GMT

Hi,
Any inputs on this issue? Anticipating your positive response on this matter.

Thanks.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

joanxie — Mon, 21 Oct 2024 06:27:40 GMT

how do you switch the streaming and what kind of errors do you get? pls share more detailed information and steps

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Mon, 21 Oct 2024 13:54:28 GMT

Hi @joanxie,

Thanks for your reply.

We are using v4l2-ctl to switch between formats.

For example,
After boot, first if we try to stream RAW10 format as given below, the stream works.

# v4l2-ctl -d 3 -v width=4208,height=3124,pixelformat='BA10' --stream-mmap

[ 56.755600] streamdb[0]=0, mode=8
[ 56.755618] streamdb[1]=1, mode=8
[ 56.758971] streamdb[2]=2, mode=8
[ 56.762301] streamdb[3]=3, mode=8
[ 56.765634] streamdb[4]=4, mode=8
[ 56.768962] streamdb[5]=5, mode=8
[ 56.772291] streamdb[6]=6, mode=8
[ 56.775607] streamdb[7]=7, mode=8
[ 56.778934] streamdb[8]=8, mode=8
[ 56.782265] Index = 0x0008 , format = 0x30314142, width = 4208, height = 3124, frate num = 20
[ 56.834770] input fmt BA10
[ 56.837510] output fmt BA10
<<<<<<<<<<<<<<<<<<<<<< 20.00 fps
<<<<<<<<<<<<<<<<<<<< 20.00 fps
<<<<<<<<<<<<<<<<<<<< 20.00 fps
<<<<<<<<<<<<<<<<<<<< 20.00 fps
<<<<<<<^C
root@ucm-imx8m-plus:~#

Then, I tried to stream the UYVY format as given below. But the system sets all the configuration in the platform and camera but stops without streaming. There is frame done interrupt in the platform.

# v4l2-ctl -d 3 -v width=4192,height=3120,pixelformat='UYVY' --stream-mmap
[ 96.384454] streamdb[0]=0, mode=4
[ 96.384471] streamdb[1]=1, mode=4
[ 96.387840] streamdb[2]=2, mode=4
[ 96.391173] streamdb[3]=3, mode=4
[ 96.394504] streamdb[4]=4, mode=4
[ 96.397832] Index = 0x0004 , format = 0x59565955, width = 4192, height = 3120, frate num = 13
[ 96.463186] bypass csc
[ 96.465562] input fmt YUV4
[ 96.468283] output fmt UYVY

The platform gives no error. But it is not able to receive the frames properly during the format switch. We have confirmed on the camera side that the requested format is configured and MIPI activity is proper in the camera hardware.

Kindly let us know in case any other additional information is needed.

Thanks.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

joanxie — Tue, 22 Oct 2024 08:07:35 GMT

if you boot up the board, firstly you test 'UYVY' , is it ok? the same issue when you capture raw10 after uyvy?

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Tue, 22 Oct 2024 11:17:17 GMT

Hi @joanxie,

Yes. After the bootup if we first test the UYVY format. It works properly as given below.

# v4l2-ctl -d 3 -v width=4192,height=3120,pixelformat='UYVY' --stream-mmap
[ 47.987726] streamdb[0]=0, mode=4
[ 47.987743] streamdb[1]=1, mode=4
[ 47.991104] streamdb[2]=2, mode=4
[ 47.994461] streamdb[3]=3, mode=4
[ 47.997801] streamdb[4]=4, mode=4
[ 48.001133] Index = 0x0004 , format = 0x59565955, width = 4192, height = 3120, frate num = 13
[ 48.066036] bypass csc
[ 48.068416] input fmt YUV4
[ 48.071127] output fmt UYVY
<<<<<<<<<<<<<<< 13.00 fps
<<<<<<<<<<<<< 13.00 fps
<<<<<<<<<<<<< 13.00 fps
<<<<<<<<<<<<< 13.00 fps
<<<<<<<<<<<<< 13.00 fps
<<<<^C
root@ucm-imx8m-plus:~#

After that, if we try RAW10 the stream not working as given below.

# v4l2-ctl -d 3 -v width=4208,height=3124,pixelformat='BA10' --stream-mmap
[ 79.250902] streamdb[0]=0, mode=8
[ 79.250920] streamdb[1]=1, mode=8
[ 79.254250] streamdb[2]=2, mode=8
[ 79.257590] streamdb[3]=3, mode=8
[ 79.260929] streamdb[4]=4, mode=8
[ 79.264264] streamdb[5]=5, mode=8
[ 79.267595] streamdb[6]=6, mode=8
[ 79.270923] streamdb[7]=7, mode=8
[ 79.274242] streamdb[8]=8, mode=8
[ 79.277571] Index = 0x0008 , format = 0x30314142, width = 4208, height = 3124, frate num = 20
[ 79.330466] input fmt BA10
[ 79.333186] output fmt BA10

Basically, only one format is working in the iMX8MPlus. Switching between the format is not working. Is there any reason for this issue and any patches available for this issue?

From our debugging, all the configuration for the format in the platform drivers are correctly setting as in the working case. But system couldn't receive the frames after the format switching.

Thanks.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

CunningJaguar — Tue, 22 Oct 2024 12:04:17 GMT

It sounds like you're troubleshooting an issue with the board’s video output. When you boot it up and test 'UYVY,' you want to make sure everything is functioning correctly before moving on to capturing in raw10 format. If you're experiencing the same problem with the raw10 capture as with the UYVY test, it could indicate a deeper issue, possibly with the board’s hardware or the way it's processing those formats. Have you checked the connections and ensured that all drivers and firmware are up to date? It might also help to look at any error messages or logs that could give more insight into what’s going wrong.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

joanxie — Wed, 23 Oct 2024 01:41:10 GMT

what bsp do you use? it seems you change the source code by yourself, how about testing the default bsp? I tested on 6.6.23 bsp with os08a20 camera, I couldn't reproduce this issue

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Wed, 23 Oct 2024 05:46:58 GMT

Hi @joanxie,

We are checking with the following BSP. But please note that we observe the same behavior even in the 5.10 and 5.15 kernel version.

BSP Details:
Kernel - v6.1.55.

Yocto - Mickledore

Please find our additional kernel changes to support UYVY and BA10 format in the attachment (ar1335_imx8mplus_kernel.patch).

Kindly share your suggestion to resolve this issue.

Thanks.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

joanxie — Wed, 23 Oct 2024 06:11:42 GMT

could you reproduce this issue with default format? I don't have this camera to reproduce this, I need to know if this issue is related to the camera or not, could you test the default format again?

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Wed, 23 Oct 2024 11:32:17 GMT

Hi @joanxie ,

As per our testing, the camera is correctly configured with required format and the MIPI activity is proper.

We suspect on the platform couldn't sample the data properly during the format switch.

Can you please explain, what do you mean by default format and how to test it?

Thanks.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

joanxie — Thu, 24 Oct 2024 02:46:11 GMT

the format you test is add by yourself, I mean you use the default format, not your own add format, you can use --list-formats to check all of format your camera support, then test the default one(not the raw10 you add), because I couldn't reproduce this on my side, I don't know if this issue is related to your camera driver or nxp bsp source code, if could, could you test the camera the bsp support as default? to check if this issue is related to your bsp or your camera

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

Olivia — Sat, 26 Oct 2024 10:13:29 GMT

Hi Joanxie,
I am using v4l2-ctl command to switch between the formats, I am able switch between Y8 to Y10 and Y10 to Y8. But I am not able to switch to Y12 from (Y8 and Y10) and vice-versa.

Below command I had used to switch from Y8 to Y10 and Y10 to Y12. After switching to Y12 format I am not getting Frames.

v4l2-ctl --device=/dev/video3 --set-fmt-video=pixelformat=GREY --stream-mmap --stream-count=20 --stream-to=raw_video_output.raw
<<<<<<<<<< 8.16 fps
<<<<<<<< 8.16 fps
<<
root@imx8mp-var-dart:~# v4l2-ctl --device=/dev/video3 --set-fmt-video=pixelformat="Y10 " --stream-mmap --stream-count=20 --stream-to=raw_video_output.raw
<<<<<<<<< 9.39 fps, dropped buffers: 2
<<<<<<< 9.39 fps, dropped buffers: 2
<<<<
root@imx8mp-var-dart:~# v4l2-ctl --device=/dev/video3 --set-fmt-video=pixelformat="Y12 " --stream-mmap --stream-count=20 --stream-to=raw_video_output.raw

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Mon, 28 Oct 2024 10:53:09 GMT

Hi @joanxie,

As per our previous discussion, I have added default supported formats (YUYV and RGB888) in the camera and tried the switching between the formats.

There are few kernel modifications required to properly set the required pixelformat in the sensor driver. Since the current iMX8MPlus kernel by default sets MEDIA_BUS_FMT_UYVY8_2X8 as the format code in the "drivers/staging/media/imx/imx8-isi-cap.c" file. Please find the kernel changes done in the attached patch.

The camera stream stops when switching between the formats same as before.

Case 1: Launching RGB888 first after the boot and then YUYV format.

root@imx8mp-var-dart:~/YUV_RGB_Binaries# v4l2-ctl -d 3 -v width=640,height=480,pixelformat='RGB3' --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 113.68 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<^C
root@imx8mp-var-dart:~/YUV_RGB_Binaries#
root@imx8mp-var-dart:~/YUV_RGB_Binaries#
root@imx8mp-var-dart:~/YUV_RGB_Binaries# v4l2-ctl -d 3 -v width=640,height=480,pixelformat='YUYV' --stream-mmap

Case 2: Launching YUYV format first after the boot and then YUYV format.

root@imx8mp-var-dart:~# v4l2-ctl -d 3 -v width=1280,height=720,pixelformat='YUYV' --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 75.40 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 75.40 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 75.40 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<^C
root@imx8mp-var-dart:~#
root@imx8mp-var-dart:~# v4l2-ctl -d 3 -v width=640,height=480,pixelformat='RGB3' --stream-mmap

Kindly check this and let us know how to resolve this behavior?

Thanks.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

joanxie — Tue, 29 Oct 2024 07:00:33 GMT

I'm taking my annual leave this week, I will check this when I come back to office, it seems this issue is related to your own code, I need double check it

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Tue, 29 Oct 2024 07:54:47 GMT

Hi @joanxie,

Sure, We will wait for your reply.

Please find the missed patch in the previous message. (nxp_imx8mplus_two_format_kernel.patch). Kindly check this and share your suggestion.

Thanks.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

Olivia — Tue, 29 Oct 2024 08:28:46 GMT

I will await your response once you return to the office. Please let me know if you need any additional information from my end to help with the review.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Mon, 04 Nov 2024 05:40:33 GMT

Hi @joanxie,

Any update on the issue?

Thanks.

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

joanxie — Tue, 05 Nov 2024 05:45:16 GMT

sorry for my delay, I've taken annual leave recently, I never tested raw12 on imx8mp before, you can check your MIPI_CSIS_ISPCFG_ALIGN_32BIT and MIPI_CSIS_CMN_CTRL_HDR_MODE, besides of these, you also can check pixel mode in your driver, I don't know if your raw12 works or not, if not, you can refer to the link as below, I also suggest that you can add more debug information to check where stuck

https://community.nxp.com/t5/i-MX-Processors/Confusion-of-MIPI-CSI2-ISI-drivers-for-IMX8MP/td-p/1958867

Re: Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

koilarulraj — Thu, 07 Nov 2024 09:10:54 GMT

Hi @joanxie,

We tried to switch between (MEDIA_BUS_FMT_SGRBG10_1X10 & MEDIA_BUS_FMT_SGRBG8_1X8) With these two formats, we can able to properly switch between the streaming. But when we tried to switch to UYVY format, the stream stops.

When debugging the drivers/staging/media/imx/imx8-mipi-csi2-sam.c driver code, In the mipi_csis_imx8mp_phy_reset function, UYVY related media bus formats are not handled.

Is this something related to the issue?

What is the PIXEL MODE value expected for the UYVY, SGRBG10, SGRBG8 formats?

Whether the imx8-mipi-csi2-sam.c driver is written to handle multiple format support?

Kindly let us know your thoughts to resolve this issue.

Thanks.