iMX8MP ISP Debayering Issue with IMX327

[stefan_wenninge]

Hello, we are currently trying to integrate our NXP i.MX8MP’s ISP into the camera pipeline of an IMX327 image sensor. The drivers implementation is based upon a driver example for the IMX219 image sensor.

This example originates from NXP’s AN13963: i.MX 8M Plus IMX219 Sensor. Sadly the original link to the related Github repository is no longer valid. Luckily a community member made a backup of the original repo, so that all the required sources are available.

Using this example (and others scattered around the NXP Community forums) we were able to implement a first revision of our IMX327 ISP driver, build it (using Yocto) and deploy it onto a development board.

Issue

The general image pipeline seems to be up and running with this driver. There is clearly correct image data from the sensor in the resulting images. However, the image stream deteriorates quite significantly over the span of a few images. After one or two seconds there seems to only be random noise in all color channels left.

Example: video_test

./video_test -w 1920 -h 1080 -f RAW10 -t file -m 0 -d 2

The following images show the first 4 frames obtained via video_test with the -t file output type. The original .raw files are contained in the attachments (see raws.zip).

video_test_image_1.pngvideo_test_image_2.png

video_test_image_3.pngvideo_test_image_4.png

The first image (video_test_image_1.png) does show a noisy but relatively clear image of the captured scene. The second image is already unrecognizable and from the third image onwards there is only random noise left.

Example: gstreamer

gst-launch-1.0 -v v4l2src device=/dev/video2 ! "video/x-raw,format=YUY2,width=1920,height=1080" ! queue ! imxvideoconvert_g2d ! ximagesink

The attached video gstreamer_video.mp4 shows the start of the gstreamer stream. The quick deterioration can be observed there aswell.

The following screenshots from the video show the deterioration frame by frame:

gstreamer_image_1.jpggstreamer_image_2.jpggstreamer_image_3.jpggstreamer_image_4.jpggstreamer_image_5.jpggstreamer_image_6.jpggstreamer_image_7.jpg

Theory

It appears to me that there is an issue during the debayering phase in the ISP. The sensors color information seems to not be correctly processed by the ISP.

My assumption is, that some part of the ISP pipeline is not configured correctly yet. There is a high chance of that being the case since all of the sensor specific configuration data was copied over from the IMX219. The idea was to first get the ISP pipeline running somehow using the IMX219’s config data and to then tune the image quality via the calibration XML file.

Tuning via vvext

A first attempt at improving the image quality using the vvext tool proved only somewhat successful. No clear detail can be made visible, only rough outlines:

gstreamer_image_optimized.png

vvext settings:

• FPS: 1

• BLS: 0, 0, 0, 0

• DEMOSAIC: normal

• DPCC: enabled

• HDR: 3, 16.0

• DWE, AEC, AWB, AF, LSC, CPROC, GAMMA, FILTER, CAC, CNR, DPF and WDR3 disabled

After these improvements it still looks like there is something wrong with the debayering process.

Reference

As a reference, here is an image shot with a Google Pixel 7 showing the same scene:

pixel7_reference_image.jpg

Reference ISI

The issue must stem from a software or configuration issue, since the hardware can be proven to work. When receiving the image data using the ISI, clear and correct looking images can be obtained (see isi_reference_image.raw in raws.zip

isi_reference_image.pngZooming in, one can see that the image now contains the bayer pattern, since it has not been debayered by the ISP:

isi_reference_image_zoomed.png

Logs

For completeness, the attached zip logs.zip contains all the logs produced by the example applications.

Contents:

• video_test application
  • Log video_test: video_test_log.txt
  • Log start_isp.sh: start_isp_log_video_test.txt
  • Log kernel: kern_log_video_test.txt
• gstreamer application
  • Log gstreamer: gstreamer_log.txt
  • Log start_isp.sh: start_isp_log_gstreamer.txt
  • Log kernel: kern_log_gstreamer.txt
• media-ctl pipeline
  • ISP
    • /dev/media0: media_ctl_print_media0_isp.txt
    • /dev/media1: media_ctl_print_media1_isp.txt
  • ISI
    • /dev/media0: media_ctl_print_media0_isi.txt

Question

Has anyone seen this kind of behaviour before? Any hints or pointers as to why the debayering seems to be failing would be greatly appreciated.

If there is any additional information I can provide to aid in finding this issue, please let me know.

[stefan_wenninge · Accepted solution]

Update: I was able to solve the issue, or at least identify it. The issue was the ISP trying to set the exposure and gain. Once I disabled it (comment out IOCTLs VVSENSORIOC_S_EXP, VVSENSORIOC_S_VSEXP, VVSENSORIOC_S_GAIN and VVSENSORIOC_S_VSGAIN) I received proper images via gstreamer:

I assume the calibration data in the .xml specifies a target value for the image quality (e.g. contrast level or median brightness). The ISP then presumably tries to optimze the image towards this target state using exposure and gain.

However I do not yet have proper calibration data for the IMX327 sensor. The calibration data I used was copied over from the IMX219 example of AN13963. Therefore the target value probably does not make sense for the IMX327 sensor.

The ISP then tries to reach an unreachable state via the setting of exposure and gain making the image quality and noise level increasingly worse.

Of course the proper fix for this problem is to produce/obtain sensor specific calibration data. That way the target state fits the IMX327 sensor.

[adde_ado]

Hi Stefan, any chance you could share your imx327_mipi.c implementation or at least help me clarify some questions?

Best regards,
Adde

[stefan_wenninge]

Hi Adde, unfortunately I am not at liberty to share the driver externally. But feel free to send me your questions and I will try to answer them.

Best regards,
Stefan

[adde_ado]

Hi Stefan,

I have sent you a private message.

Best regards,
Adde

[joanxie]

I have mailed to you, pls check it

[khang_letruong]

Dear @joanxie ,

I got following color issue with the 2nd camera pipeline, even if it was the only launched pipeline :

The good image (from the fist pipeline) should be as below :

Have you ever noticed similar issue? Knowing that we are using the LF6.1.36-2.1.0 release.

Thanks and best regards,
Khang