Camera sensor hi846 MIPI CSI for IMX8MM: Image issue

[shixuanlin-ddcx]

Dear NXP Support Team,

I am writing to seek urgent assistance regarding critical issues I am experiencing with an i.MX8MM platform running kernel version 5.15. The problems involve video capture and real-time display using GStreamer pipelines with a Hi846 camera module, and I would greatly appreciate your expertise in resolving them.

• Hardware and Software Setup:

           Platform: i.MX8MM

           Kernel: 5.15

           Camera: Hi846 with minimum resolution of 1280x720

• Successful Data Capture:

            I can capture raw data using the following command: v4l2-ctl -d /dev/video0 --set-fmt-video width=1280,height=720,pixelformat=BA10 --stream-mmap --stream-count=10 --stream-to=test.raw

            I can also capture a single PNG image with this GStreamer command: gst-launch-1.0 -vvv v4l2src device=/dev/video0 num-buffers=1 ! "video/x-bayer,width=1280,height=720,framerate=(fraction)15/1,format=(string)grbg" ! bayer2rgb ! pngenc ! filesink location=test.png

• Real-Time Display Issues: 

           However, when attempting real-time display with the following pipeline: gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-bayer,format=grbg,width=1280,height=720,framerate=15/1 ! queue ! bayer2rgb ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,pixel-aspect-ratio=1/1 ! glimagesink

           I encounter significant lag and the following warning: 

WARNING: from element /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink: A lot of buffers are being dropped.
Additional debug info:
../git/libs/gst/base/gstbasesink.c(3143): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink:
There may be a timestamping problem, or this computer is too slow.

           The display is choppy, indicating performance issues.

• High CPU Usage: 

         During the real-time display attempt, CPU usage spikes to nearly 100%, as shown in the top output:

I have tried adding queue elements and adjusting frame rates, but the issue persists

• Request for Assistance

This high CPU usage and frame dropping are severely impacting real-time performance, which is critical for my application. I suspect the issue may be related to inefficient buffer handling, lack of hardware acceleration, or a configuration mismatch with the i.MX8MM’s GPU/VPU capabilities. Could you please:

1. Provide guidance on optimizing the GStreamer pipeline to leverage hardware acceleration ?
2. Suggest kernel or driver patches to improve DMA buffer handling or reduce CPU load?
3. Recommend debugging steps to identify the exact bottleneck (e.g., enabling specific GStreamer debug levels or kernel logs)?

I am available to provide additional logs, such as GST_DEBUG=3 output or kernel traces, upon request. This is an urgent matter, and your prompt support would be greatly appreciated.

Thank you for your attention to this issue.

Best regards,

[Manuel_Salas]

Hello @shixuanlin-ddcx 

I hope you are doing very well.

Can your camera output YUV (UYVY/NV12)? If yes, you can skip Bayer decode and go straight to display.

Please try the below to see the available formats:

v4l2-ctl --list-formats-ext -d /dev/video0

Also, you can try:

gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-raw,width=1280,height=720,format=NV12 ! \
  queue ! waylandsink sync=false

That should improve because bayer2rgb and videoconvert are CPU-bound and not hardware-accelerated.

I hope this information can helps to you.

Best regards,

Salas.

[shixuanlin-ddcx]

Dear NXP Support Team,

Thank you for your prompt response and helpful suggestions. 

I have checked the available formats for my camera (HI846) using the command you provided:v4l2-ctl --list-formats-ext -d /dev/video0：

The output shows that my camera only supports the Bayer format, specifically 'BA10' (10-bit Bayer GRGR/BGBG), with the following resolutions:

mx6s_captuere.c: mbus_code 300a
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

[0]: 'BA10' (10-bit Bayer GRGR/BGBG)
Size: Discrete 3264x2448
Size: Discrete 640x480
Size: Discrete 1280x720
Size: Discrete 1632x1224

Since the camera does not support YUV formats like UYVY or NV12, I am unable to skip the Bayer decoding step as suggested.

Given that bayer2rgb and videoconvert are CPU-bound and not hardware-accelerated, could this be the primary reason for the performance issues I am experiencing? Additionally, I would like to confirm if the Bayer decoding process might be contributing to any potential issues. Also, since /dev/media0 has not been generated, could its creation have any impact on the current behavior?

I would greatly appreciate your insights on these matters and any further suggestions to improve performance with the Bayer format.

Looking forward to your reply.

Best regards

[Manuel_Salas · Accepted solution]

Hello @shixuanlin-ddcx 

The bayer2rgb and videoconvert elements in your GStreamer pipeline put a heavy load on the CPU because they aren't designed to use hardware acceleration. As a result, they consume a lot of processing power, which leads to high CPU usage (around 100%) and causes dropped frames during real-time preview.

Please try to reduce resolution:

gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-bayer,format=grbg,width=1280,height=720,framerate=15/1 ! \
  bayer2rgb ! \
  videoscale method=1 ! \
  video/x-raw,width=640,height=360 ! \
  videoconvert ! \
  autovideosink

Or:

gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-bayer,format=grbg,width=1280,height=720,framerate=15/1 ! \
  bayer2rgb ! \
  videoscale method=1 ! \
  video/x-raw,width=640,height=360 ! \
  videoconvert ! \
  waylandsink sync=false

Best regards,

Salas.

[shixuanlin-ddcx]

Thank you very much for your reply. The command you provided is very effective and no more frame drops.

gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-bayer,format=grbg,width=1280,height=720,framerate=15/1 ! \
  bayer2rgb ! \
  videoscale method=1 ! \
  video/x-raw,width=640,height=360 ! \
  videoconvert ! \
  waylandsink sync=false