i.MX8QXP capture raw(bayer) data and debayer

Document created by PengFushi Employee on Dec 1, 2019Last modified by PengFushi Employee on Dec 2, 2019
Version 2Show Document
  • View in full screen mode

This doc show:

on i.MX8QXP MEK board, configure ov5640 sensor(parallel interface) output 5MP(2592x1944) RAW(Bayer) data at 15fps, and Parallel Capture Subsystem and Image Sensor Interface capture RAW RGB data, and i.MX8QXP GPU debayer RAW data then display image.

HW: i.MX8QXP MEK board, MEK base board (to place the parallel camera), ov5640 sensor.

SW: Linux 4.14.98_2.0.0 BSP, and patches in this doc.

 

  1. Configure at camera sensor side

A Bayer filter is a color filter array (CFA) for arranging RGB color filters on a square grid of photosensors. The filter pattern is 50% green, 25% red and 25% blue, hence is also called BGGR, RGBG ,GRGB, or RGGB.

The ov5640 has an image array capable of operating at up to 15 fps in 5 megapixel (2592x1944) resolution. OV5640 support output formats: RAW(Bayer), RGB565/555/444,CCIR656, YUV422/420, YCbCr422, and compression.

To make ov5640 output 5MP RAW data at 15fps, check my kernel patch imx8-ov5640-raw-capture-driver-4.14.98_2.0.0.diff which apply on i.MX Linux 4.14.98_2.0.0 BSP kernel code.

Parallel interface ov5640, use ov5640_raw_setting[] array of drivers/media/platform/imx8/ov5640_v3.c. This register setting is come from ov5640 software application note and data sheet.

 

  1. Configure at i.MX8QXP side

The Parallel Capture Subsystem consists of the Parallel Capture Interface (BT 656) and associated peripherals. It interfaces to the Parallel CSI sensor. This allows for up to 24 RGB data bits in parallel or for RGB components on consecutive clocks (up to 10-bit color depth). The formats supported are RGB, RAW and YUV 422.

Below is Parallel Capture Subsystem diagram:

For RAW format data, CSI_CTRL_REG of Parallel Capture Subsystem need configured as my patch, otherwise found cannot get correct data.

 

The multiple input sources (MIPI CSI, Parallel Capture) captures the pixel data and feeds it to the ISI. The ISI is responsible for capturing and pre-processing the pixel data from multiple input sources and storing them into the memory.

Below is ISI diagram:

 

For RAW format data, it should be bypass any processing pipeline of ISI, just use ISI to save it to memory.

 

  1. Capture test code

To capture the RAW data and save it to file, check my patch imx8_ov5640_raw_captupre_test_4.14.98_2.0.0_ga.diff which apply on i.MX Linux 4.14.98_2.0.0 BSP unit test code.

Note the usage is:

./imx8_cap.out -of -cam 1 -fr 15 -fmt BA81 -ow 2592 -oh 1944 -num 100

 

  1. Display RAW data

The RAW data cannot be displayed directly, debayer process is needed to get complete red, green, blue color for each pixel.

The debayer process if run on CPU, will cost much CPU time.

To save CPU time, debayer could done by GPU.

The method is, captured RAW data upload to GPU as texture , then GPU will do the debayer, then full color of each pixel will be got, then display it.

To upload RAW camera data to GPU with zero memory copy, i will use i.MX8QXP GPU extension GL_VIV_direct_texture. It create a texture with direct access support. API glTexDirectVIVMap,  which support mapping a user space memory or a physical address into the texture surface.

The API glTexDirectVIVMap need logic and physical address of data buffer, so i will allocate data buffer from g2d lib, it is dma-buffer also get logic/physical address of buffer, then queue it as DMABUF to v4l2 capture driver, after dequeue got RAW camera data, pass it to GPU for debayer.

GPU side, I will use OpenGL shader code from "Efficient, High-Quality Bayer Demosaic Filtering on GPUs".

Check my patch imx8_debayer-gpusdk-5.3.0.diff which apply on i.MX GPU SDK 5.3.0 code.

Note, here i only do is debayer, no extra process.

 

  1. Known issue

One thing is ov5640 output 5MP at 15fps, compare with output 5MP at 5fps, there are more noise of camera data at 15fps case. My debug found is, this noise seems come from ov5640 itself.

 

Reference:

a>https://www.nxp.com/webapp/Download?colCode=IMX8DQXPRM

b>https://www.nxp.com/webapp/Download?colCode=L4.14.98_2.0.0_MX8QXP&appType=license

c>https://github.com/NXPmicro/gtec-demo-framework

d>ov5640 data sheet

e>ov5640 software application note

f>Efficient, High-Quality Bayer Demosaic Filtering on GPUs

https://www.semanticscholar.org/paper/Efficient%2C-High-Quality-Bayer-Demosaic-Filtering-on-McGuire/088a2f47b7ab99c78d41623bdfaf4acdb02358fb

3 people found this helpful

Outcomes