Switching the formats in the camera fails to launch streaming in the iMX8Mplus platform

[koilarulraj]

HI,

We have supported UYVY, SRGB10 formats from the camera and using iMX8MPlus kit for the evaluation.

After booting of the kit, if we first launch either UYVY or SRGB10, the stream works without any issues in the format. But when we try to switch between the format, the stream stops.

We have debugged the camera MIPI signals. It is correctly configuring with the required format and MIPI signal activity is proper in the camera.

But the iMX8Mplus platform not supporting the switching between the different formats. Is there any patches to resolve this issue?

We are using following kernel.
kernel : v6.1.55
Yocto : mickledore

Thanks.

[koilarulraj]

Hi @joanxie,

As per our previous discussion, I have added default supported formats (YUYV and RGB888) in the camera and tried the switching between the formats.

There are few kernel modifications required to properly set the required pixelformat in the sensor driver. Since the current iMX8MPlus kernel by default sets MEDIA_BUS_FMT_UYVY8_2X8 as the format code in the "drivers/staging/media/imx/imx8-isi-cap.c" file. Please find the kernel changes done in the attached patch.

The camera stream stops when switching between the formats same as before.

Case 1: Launching RGB888 first after the boot and then YUYV format.

root@imx8mp-var-dart:~/YUV_RGB_Binaries# v4l2-ctl -d 3 -v width=640,height=480,pixelformat='RGB3' --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 113.68 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<^C
root@imx8mp-var-dart:~/YUV_RGB_Binaries#
root@imx8mp-var-dart:~/YUV_RGB_Binaries#
root@imx8mp-var-dart:~/YUV_RGB_Binaries# v4l2-ctl -d 3 -v width=640,height=480,pixelformat='YUYV' --stream-mmap

Case 2: Launching YUYV format first after the boot and then YUYV format.

root@imx8mp-var-dart:~# v4l2-ctl -d 3 -v width=1280,height=720,pixelformat='YUYV' --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 75.40 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 75.40 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 75.40 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<^C
root@imx8mp-var-dart:~#
root@imx8mp-var-dart:~# v4l2-ctl -d 3 -v width=640,height=480,pixelformat='RGB3' --stream-mmap

Kindly check this and let us know how to resolve this behavior?

Thanks.

[koilarulraj]

Hi @joanxie ,

As per our testing, the camera is correctly configured with required format and the MIPI activity is proper.

We suspect on the platform couldn't sample the data properly during the format switch.

Can you please explain, what do you mean by default format and how to test it?

Thanks.

[joanxie]

the format you test is add by yourself, I mean you use the default format, not your own add format, you can use --list-formats to check all of format your camera support, then test the default one(not the raw10 you add), because I couldn't reproduce this on my side, I don't know if this issue is related to your camera driver or nxp bsp  source code, if could, could you test the camera the bsp support as default? to check if this issue is related to your bsp or your camera

[koilarulraj]

Hi @joanxie,

We tried to switch between (MEDIA_BUS_FMT_SGRBG10_1X10 & MEDIA_BUS_FMT_SGRBG8_1X8) With these two formats, we can able to properly switch between the streaming. But when we tried to switch to UYVY format,  the stream stops.

When debugging the drivers/staging/media/imx/imx8-mipi-csi2-sam.c driver code, In the mipi_csis_imx8mp_phy_reset function, UYVY related media bus formats are not handled.

Is this something related to the issue?

What is the PIXEL MODE value expected for the UYVY, SGRBG10, SGRBG8 formats?

Whether the imx8-mipi-csi2-sam.c driver is written to handle multiple format support?

Kindly let us know your thoughts to resolve this issue.

Thanks.

[joanxie]

"When debugging the drivers/staging/media/imx/imx8-mipi-csi2-sam.c driver code, In the mipi_csis_imx8mp_phy_reset function, UYVY related media bus formats are not handled."

> what do you mean media bus are not handled? do you  mind adding more debug information in this function and share more log? since I couldn't reproduce this issue and your logfile is too limit to find the root cause

[koilarulraj]

Hi @joanxie ,

Thanks for the reply. please find the reply below.

what do you mean media bus are not handled?

please find the attached image where in the "mipi_csis_imx8mp_phy_reset" function in the kernel platform driver(drivers/staging/media/imx/imx8-mipi-csi2-sam.c), the format "MEDIA_BUS_FMT_UYVY8_2X8" not handled. This phy_reset is getting called whenever we change the format & resolutions. And this function handles RAW8,10 and 12 format settings in the phy reset. So suspected whether this could be the reason.

do you mind adding more debug information in this function and share more log?

Please find the attached logs with more information.

Working_UUVY.txt - Log when streaming UYVY after the fresh boot.

NotWorking_UUVY.txt - Log after switching to other format and then retry UYVY.

Thanks.

[joanxie]

it seems stuck here, couldn't get proper signal

https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/drivers/media/common/videobuf2/videobuf2-core.c#L...

for more detailed information, refer to the function

https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/include/linux/wait.h#L496

maybe you can add more debug information to trace, how about using gstreamer or unit test to change the format? still the same issue?

[koilarulraj]

Hi @joanxie,

Please find the replies below.

maybe you can add more debug information to trace,

Please find the attachments with more debug logs enabled from the kernel (CONFIG_DYNAMIC_DEBUG) and v4l2 core.

how about using gstreamer or unit test to change the format? still the same issue?

Yes. The format switch issue occurs even with Gstremer and unit test when changing the format.

The v4l2 layer not receiving the frame event. Seems like something failing at the CSI platform layer which blocks the system to properly receive the data. Kindly check the logs and share your suggestions.

Thanks.

[koilarulraj]

Hi @joanxie ,

Did you get chance to review the above logs with Gstreamer commands?

Suggestions from your end would be helpful to proceed with the debugging.

Thanks.

[koilarulraj]

Hi @joanxie,

Any inputs on this topic. We would really appreciate any debugging technique to gather more details on the issue.

Thanks.

[joanxie]

sorry I was handling a urgent case before, could you tell me what do the each logfiles stand for? they all work under yuv format, right? 

[koilarulraj]

Hi @joanxie,

Sure. Please find the explanation for each log file.

working_uyvy_log_with_gstreamer.txt

     a) Logs when streaming UYVY format after fresh boot.

not_working_jpeg_format_log_with_gstreamer.txt

     a) Log when switching the stream to MJPG format from UYVY format.

     b) Now stream blocks without any error in the dmesg. And no frame ready interrupt received in the platform drivers.

not_working_uyvy_format_after_switch_from_jpeg_format_log_with_gstreamer.txt

     a) During the not working condition, stop the MJPG stream and again tried launching UYVY format. There is no stream received in the platform.

The stream fails when switch from or switch to UYVY format in the iMX8Mplus platform. Looking forward to your response to resolve this issue.

Thanks.  

[joanxie]

I remembered you couldn't switch between raw data with yuv data, now you test between yuv with mjpeg? anyway, I checked your not_working_jpeg_format_log_with_gstreamer, CSIS_DPHYSTATUS[20]: 0x00000001, which means the clock is under stop state, you need check if this bit is always be set in stop state, and when you capture, try to use oscilloscope to check if the camera work correctly after you switch

[koilarulraj]

Hi @joanxie,

As per the ISI_OUT_FMT definition in the TRM (image in the attachment), YUYV format is stored interleaved packed bytes. Whereas RAW formats are stored word type. Will this cause any problem when switching between the formats in memory level?. Because, we observe issues only when switching between YUYV format.

We aware that, using ISP interface switching is working with os08a20 camera. But is there any reference for cameras which only use the ISI interface to properly switch between the formats.

Looking forward to your response.

[koilarulraj]

Hi @joanxie,

We are checking with the following BSP. But please note that we observe the same behavior even in the 5.10 and 5.15 kernel version.

BSP Details:
Kernel - v6.1.55.

Yocto - Mickledore

Please find our additional kernel changes to support UYVY and BA10 format in the attachment (ar1335_imx8mplus_kernel.patch).

Kindly share your suggestion to resolve this issue.

Thanks.

[joanxie]

could you reproduce this issue with default format? I don't have this camera to reproduce this, I need to know if this issue is related to the camera or not, could you test the default format again?

[CunningJaguar]

It sounds like you're troubleshooting an issue with the board’s video output. When you boot it up and test 'UYVY,' you want to make sure everything is functioning correctly before moving on to capturing in raw10 format. If you're experiencing the same problem with the raw10 capture as with the UYVY test, it could indicate a deeper issue, possibly with the board’s hardware or the way it's processing those formats. Have you checked the connections and ensured that all drivers and firmware are up to date? It might also help to look at any error messages or logs that could give more insight into what’s going wrong.

[koilarulraj]

Hi @joanxie,

Yes. After the bootup if we first test the UYVY format. It works properly as given below.

# v4l2-ctl -d 3 -v width=4192,height=3120,pixelformat='UYVY' --stream-mmap
[ 47.987726] streamdb[0]=0, mode=4
[ 47.987743] streamdb[1]=1, mode=4
[ 47.991104] streamdb[2]=2, mode=4
[ 47.994461] streamdb[3]=3, mode=4
[ 47.997801] streamdb[4]=4, mode=4
[ 48.001133] Index = 0x0004 , format = 0x59565955, width = 4192, height = 3120, frate num = 13
[ 48.066036] bypass csc
[ 48.068416] input fmt YUV4
[ 48.071127] output fmt UYVY
<<<<<<<<<<<<<<< 13.00 fps
<<<<<<<<<<<<< 13.00 fps
<<<<<<<<<<<<< 13.00 fps
<<<<<<<<<<<<< 13.00 fps
<<<<<<<<<<<<< 13.00 fps
<<<<^C
root@ucm-imx8m-plus:~#

After that, if we try RAW10 the stream not working as given below.

# v4l2-ctl -d 3 -v width=4208,height=3124,pixelformat='BA10' --stream-mmap
[ 79.250902] streamdb[0]=0, mode=8
[ 79.250920] streamdb[1]=1, mode=8
[ 79.254250] streamdb[2]=2, mode=8
[ 79.257590] streamdb[3]=3, mode=8
[ 79.260929] streamdb[4]=4, mode=8
[ 79.264264] streamdb[5]=5, mode=8
[ 79.267595] streamdb[6]=6, mode=8
[ 79.270923] streamdb[7]=7, mode=8
[ 79.274242] streamdb[8]=8, mode=8
[ 79.277571] Index = 0x0008 , format = 0x30314142, width = 4208, height = 3124, frate num = 20
[ 79.330466] input fmt BA10
[ 79.333186] output fmt BA10

Basically, only one format is working in the iMX8MPlus. Switching between the format is not working. Is there any reason for this issue and any patches available for this issue?

From our debugging, all the configuration for the format in the platform drivers are correctly setting as in the working case. But system couldn't receive the frames after the format switching.

Thanks.

[joanxie]

what bsp do you use? it seems you change the source code by yourself, how about testing the default bsp? I tested on 6.6.23 bsp with os08a20 camera, I couldn't reproduce this issue

[koilarulraj]

Hi @joanxie,

Thanks for your reply.

We are using v4l2-ctl to switch between formats.

For example,
After boot, first if we try to stream RAW10 format as given below, the stream works.

# v4l2-ctl -d 3 -v width=4208,height=3124,pixelformat='BA10' --stream-mmap

[ 56.755600] streamdb[0]=0, mode=8
[ 56.755618] streamdb[1]=1, mode=8
[ 56.758971] streamdb[2]=2, mode=8
[ 56.762301] streamdb[3]=3, mode=8
[ 56.765634] streamdb[4]=4, mode=8
[ 56.768962] streamdb[5]=5, mode=8
[ 56.772291] streamdb[6]=6, mode=8
[ 56.775607] streamdb[7]=7, mode=8
[ 56.778934] streamdb[8]=8, mode=8
[ 56.782265] Index = 0x0008 , format = 0x30314142, width = 4208, height = 3124, frate num = 20
[ 56.834770] input fmt BA10
[ 56.837510] output fmt BA10
<<<<<<<<<<<<<<<<<<<<<< 20.00 fps
<<<<<<<<<<<<<<<<<<<< 20.00 fps
<<<<<<<<<<<<<<<<<<<< 20.00 fps
<<<<<<<<<<<<<<<<<<<< 20.00 fps
<<<<<<<^C
root@ucm-imx8m-plus:~#

Then, I tried to stream the UYVY format as given below. But the system sets all the configuration in the platform and camera but stops without streaming. There is frame done interrupt in the platform.

# v4l2-ctl -d 3 -v width=4192,height=3120,pixelformat='UYVY' --stream-mmap
[ 96.384454] streamdb[0]=0, mode=4
[ 96.384471] streamdb[1]=1, mode=4
[ 96.387840] streamdb[2]=2, mode=4
[ 96.391173] streamdb[3]=3, mode=4
[ 96.394504] streamdb[4]=4, mode=4
[ 96.397832] Index = 0x0004 , format = 0x59565955, width = 4192, height = 3120, frate num = 13
[ 96.463186] bypass csc
[ 96.465562] input fmt YUV4
[ 96.468283] output fmt UYVY

The platform gives no error. But it is not able to receive the frames properly during the format switch. We have confirmed on the camera side that the requested format is configured and MIPI activity is proper in the camera hardware.

Kindly let us know in case any other additional information is needed.

Thanks.