topic Re: Camera sensor hi846 MIPI CSI for IMX8MM: Image issue in i.MX Processors

Camera sensor hi846 MIPI CSI for IMX8MM: Image issue

shixuanlin-ddcx — Wed, 21 May 2025 12:05:38 GMT

Dear NXP Support Team,

I am writing to seek urgent assistance regarding critical issues I am experiencing with an i.MX8MM platform running kernel version 5.15. The problems involve video capture and real-time display using GStreamer pipelines with a Hi846 camera module, and I would greatly appreciate your expertise in resolving them.

Hardware and Software Setup:

Platform: i.MX8MM

Kernel: 5.15

Camera: Hi846 with minimum resolution of 1280x720

Successful Data Capture:

I can capture raw data using the following command: v4l2-ctl -d /dev/video0 --set-fmt-video width=1280,height=720,pixelformat=BA10 --stream-mmap --stream-count=10 --stream-to=test.raw

I can also capture a single PNG image with this GStreamer command: gst-launch-1.0 -vvv v4l2src device=/dev/video0 num-buffers=1 ! "video/x-bayer,width=1280,height=720,framerate=(fraction)15/1,format=(string)grbg" ! bayer2rgb ! pngenc ! filesink location=test.png

Real-Time Display Issues:

However, when attempting real-time display with the following pipeline: gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-bayer,format=grbg,width=1280,height=720,framerate=15/1 ! queue ! bayer2rgb ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,pixel-aspect-ratio=1/1 ! glimagesink

I encounter significant lag and the following warning:

WARNING: from element /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink: A lot of buffers are being dropped.
Additional debug info:
../git/libs/gst/base/gstbasesink.c(3143): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink:
There may be a timestamping problem, or this computer is too slow.

The display is choppy, indicating performance issues.

High CPU Usage:

During the real-time display attempt, CPU usage spikes to nearly 100%, as shown in the top output:

I have tried adding queue elements and adjusting frame rates, but the issue persists

Request for Assistance

This high CPU usage and frame dropping are severely impacting real-time performance, which is critical for my application. I suspect the issue may be related to inefficient buffer handling, lack of hardware acceleration, or a configuration mismatch with the i.MX8MM’s GPU/VPU capabilities. Could you please:

Provide guidance on optimizing the GStreamer pipeline to leverage hardware acceleration ?
Suggest kernel or driver patches to improve DMA buffer handling or reduce CPU load?
Recommend debugging steps to identify the exact bottleneck (e.g., enabling specific GStreamer debug levels or kernel logs)?

I am available to provide additional logs, such as GST_DEBUG=3 output or kernel traces, upon request. This is an urgent matter, and your prompt support would be greatly appreciated.

Thank you for your attention to this issue.

Best regards,

Re: Camera sensor hi846 MIPI CSI for IMX8MM: Image issue

Manuel_Salas — Mon, 26 May 2025 20:46:03 GMT

Hello @shixuanlin-ddcx

I hope you are doing very well.

Can your camera output YUV (UYVY/NV12)? If yes, you can skip Bayer decode and go straight to display.

Please try the below to see the available formats:

v4l2-ctl --list-formats-ext -d /dev/video0

Also, you can try:

gst-launch-1.0 v4l2src device=/dev/video0 ! \ video/x-raw,width=1280,height=720,format=NV12 ! \ queue ! waylandsink sync=false

That should improve because bayer2rgb and videoconvert are CPU-bound and not hardware-accelerated.

I hope this information can helps to you.

Best regards,

Salas.

Re: Camera sensor hi846 MIPI CSI for IMX8MM: Image issue

shixuanlin-ddcx — Wed, 28 May 2025 01:10:13 GMT

Dear NXP Support Team,

Thank you for your prompt response and helpful suggestions.

I have checked the available formats for my camera (HI846) using the command you provided:v4l2-ctl --list-formats-ext -d /dev/video0：

The output shows that my camera only supports the Bayer format, specifically 'BA10' (10-bit Bayer GRGR/BGBG), with the following resolutions:

mx6s_captuere.c: mbus_code 300a
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

[0]: 'BA10' (10-bit Bayer GRGR/BGBG)
Size: Discrete 3264x2448
Size: Discrete 640x480
Size: Discrete 1280x720
Size: Discrete 1632x1224

Since the camera does not support YUV formats like UYVY or NV12, I am unable to skip the Bayer decoding step as suggested.

Given that bayer2rgb and videoconvert are CPU-bound and not hardware-accelerated, could this be the primary reason for the performance issues I am experiencing? Additionally, I would like to confirm if the Bayer decoding process might be contributing to any potential issues. Also, since /dev/media0 has not been generated, could its creation have any impact on the current behavior?

I would greatly appreciate your insights on these matters and any further suggestions to improve performance with the Bayer format.

Looking forward to your reply.

Best regards

Re: Camera sensor hi846 MIPI CSI for IMX8MM: Image issue

Manuel_Salas — Wed, 28 May 2025 16:52:38 GMT

Hello @shixuanlin-ddcx

The bayer2rgb and videoconvert elements in your GStreamer pipeline put a heavy load on the CPU because they aren't designed to use hardware acceleration. As a result, they consume a lot of processing power, which leads to high CPU usage (around 100%) and causes dropped frames during real-time preview.

Please try to reduce resolution:

gst-launch-1.0 v4l2src device=/dev/video0 ! \ video/x-bayer,format=grbg,width=1280,height=720,framerate=15/1 ! \ bayer2rgb ! \ videoscale method=1 ! \ video/x-raw,width=640,height=360 ! \ videoconvert ! \ autovideosink

Or:

Best regards,

Salas.

Re: Camera sensor hi846 MIPI CSI for IMX8MM: Image issue

shixuanlin-ddcx — Thu, 29 May 2025 02:07:18 GMT

Thank you very much for your reply. The command you provided is very effective and no more frame drops.

gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-bayer,format=grbg,width=1280,height=720,framerate=15/1 ! \
  bayer2rgb ! \
  videoscale method=1 ! \
  video/x-raw,width=640,height=360 ! \
  videoconvert ! \
  waylandsink sync=false