i.MX8QM CSI-2 RAW 10 Bayer Processing

[sbertrand]

Summary of support ticket regarding integration of RAW10 Bayer GRBG CSI-2 cameras on a i.MX8QM platform.

- How can RAW10 can be capture via the i.MX8QM ISI ?
- How to process the Bayer GRBG into RGB/YUV ?

CAMERA CAPTURE

To validate RAW capture, the camera stream was configured to send specific pattern in the Bayer GRBG :

G 0x0345 R 0x0123
B 0x02EF G 0x1AB

RAW16 Format

hexdump cam.raw | head
0000000 3450 1230 3450 1230 3450 1230 3450 1230
0000f00 2ef0 1ab0 2ef0 1ab0 2ef0 1ab0 2ef0 1ab0
==>> Left shift 4-bit

RAW8 Format

hexdump -C cam.raw | head
00000000 50 30 50 30 50 30 50 30 50 30 50 30 50 30 50 30 |P0P0P0P0P0P0P0P0|
00000f00 f0 b0 f0 b0 f0 b0 f0 b0 f0 b0 f0 b0 f0 b0 f0 b0 |................|
==>> Left shift 4-bit, 8-lsb kept

RAW10 Format

hexdump -C cam.raw | head
00000000 00 14 00 8c 00 14 00 8c 00 14 00 8c 00 14 00 8c |................|
*
00000f00 00 bc 00 ac 00 bc 00 ac 00 bc 00 ac 00 bc 00 ac |................|
*
==>> Left shift 4-bit with an additional 6-bit left shift

G 0xD1400 R 0x48c00
B 0xBBC00 G 0x6AC00

NXP Support Update

For all RAW data, ISI can only choose RAW16 to receive and save the data to memory.

In the reference manual: ISI register CHNL_IMG_CTRL, FORMAT field

001100b - RAW10 - 10-bit RAW data packed into 16-bit WORD with 6 LSBs waste bits

NXP Support Update

This description is wrong, and internal team has submitted a ticket to modify this. As the Note says, "bit[15-14] and bit[3-0] being padding", but ISI cannot read from bit 13. It always reads from the start of those 16 bits and length of 16. So, the first two bits and last four bits are waste bits. Not the LSBs.

This would match the note in CSI-2 Rx Controller description

The CSI data is right LSB aligned and zero padded depending on data format. When interfacing ISI, CSI data is shifted 6-bits due to ISI bits [5:0] always being zero (0bxxCSIDATAxxxxxx).

All RAW14, RAW12, RAW10,RAW8, and RAW6 video data is filled from BIT[13] to LSB, the remaining bits are zero padded. Only RAW16 input data will be saved to memory from BIT[15] to LSB.

This RAW data is shifted which makes it hard for direct processing from a Y16 or Y8 buffer.
Testing showed that some RGB and YUV may be able to capture the 8-lsb.

ISI Format 18 - YUV444_3P8P

hexdump -C cam.raw -s $((1920*1080*3)) | head -n 4
005eec00 45 23 45 23 45 23 45 23 45 23 45 23 45 23 45 23 |E#E#E#E#E#E#E#E#|
*
005ef380 ef ab ef ab ef ab ef ab ef ab ef ab ef ab ef ab |................|
*
G 0x0345 R 0x0123
B 0x02EF G 0x1AB

Above the LSB of our bayer data are there. Storing in 3 planes allows to access the data directly. This may help for RAW8 ISI capture by keeping only the relevant plane.

Testing with YUV444_3P10 only reported the 8-lsb shifted by 8 being saved, thus not able to get RAW10 this way.

BAYER Processing

Bayer is not supported natively by the i.MX8QM. NXP GPU SDK has example of Bayer conversion using the GPU :
- https://www.nxp.com/docs/en/application-note/AN12060.pdf
- https://github.com/nxp-imx/gtec-demo-framework/tree/master/DemoApps/OpenVX/SoftISP

NXP Support Feedback

If the SoC is configured for both GPU to be independent, could they be used independently to perform BAYER conversion ?
Yes. Just refer to https://www.nxp.com.cn/docs/en/user-guide/i.MX_Graphics_User's_Guide_Linux.pdf 8.3-8.4, app can get GPU ID by clGetDeviceIDs, and assign current softISP application to specificGPU.touse two GPU, it may need two softISP software running, and assign each softISP to a target GPU. Currently softISP uses Opencl, which not support multiple GPUs.

I checked with softISP owner, it is said opengles may support multiple GPUs, you can try to do your own porting.

Processing Bayer GRBG with the i.MX8 GPU may be possible with the following work
- update the soft ISP from Bayer BGGR to Bayer GRBG (for our case)
- update the soft ISP for 4-bit shifted RAW10 data (0bxxCSIDATAxxxxxx)
- enable 2 GPU for multiple concurrent buffer processing (4 camera streams in our case)
- perform capture pipeline ( ISI -> GPU -> Memory ) (GStreamer integration ?)
- hope for good performance ( 30fps ?, at least above 15fps ?)

We will look at camera with RGB/YUV support for better compatibility.

Thanks @hector_delgado for your support.
Stan