topic Re: IPU - bayer pattern conversion in i.MX Processors

IPU - bayer pattern conversion

tollerhecht — Mon, 17 Dec 2012 15:30:03 GMT

Hi,

I'm planning to use a i.mx6 in an digital imaging application. The intention is to transfer image data from a FPGA device to the i.mx6 memory using the parallel CSI ports. The format of the image data is a bayer pattern. In the documention I can see that the IPU can receive bayer pattern and generic image data. I'm not shure how the IPU really handles the bayer data and what's the difference. Does the IPU unit contain any bayer demosaicing hardware ? Is the bayer data automatically converted to RGB or YUV when transfering to memory ?

Can anybody help me ?

Thanks and best regards

Oliver

Re: IPU - bayer pattern conversion

qiang_li-mpu_se — Tue, 18 Dec 2012 03:16:12 GMT

The CSI module can support the bayer format, you need enable the CSI CPD function in register IPUx_CSIO_CPD_CTRL, and after that, you also need config the IPUx_CSIO_CPD_xxxx registers based on the bayer detail format, GBGB or GRGR.

If you want to convert the bayer data to YUV/RGB from CSI to system memory, then CSI can only handle the 8bits MSB for each R/G/B in bayer format. Just like some other YUV camera sensors, you only need add the CSI CPD related config to support the bayer image, all other codes can be re-used in this mode.

There is another method to capture the bayer image from CSI, it is the generic mode, in this mode, the bayer image can be captured into memory as it's raw data, if it is 12bits bayer image, each 12 bits R/G/B will be captured into memory in each 16bits (Only 12 bits MSB has the valid data), in this mode, the in memory bayer data can't be handled by IPU directly, only CSI module can handle the bayer data directly. You need use software to convert that bayer image to other format.

Re: IPU - bayer pattern conversion

tollerhecht — Tue, 18 Dec 2012 09:27:50 GMT

Qiang,

Thank you very much for your answer, this was very helpful.

For the CSI-1 port the IPUx_CSI1_CPD_CTRL register is filled with reserved bits. Does this work only on the CSI-0 port ? Another problem is, that the images I want to convert have a header that contain some special information (timestamp, etc.). Is there a way to prevent the header data from being modified, when using the bayer conversion (skip) ?

Thanks and best regards,

Oliver

Re: IPU - bayer pattern conversion

qiang_li-mpu_se — Wed, 19 Dec 2012 01:37:44 GMT

If there's header data in bayer frames, then you can't use the CPD function, for such use case, you can only use the generic mode to capture the data, in this mode, all received data will be saved to memory without process, and IPU can't convert the bayer frames to RGB/YUV on the fly, you must use software to handle it.

Re: IPU - bayer pattern conversion

tollerhecht — Wed, 19 Dec 2012 16:27:16 GMT

Qiang,

thanks.
The register description of IPUx_CSI1_CPD_CTRL in the reference manual shows only reserved bits (page 3236). Does this mean that the bayer conversion works only on CSI-0 ?

Thanks and best regards,

Oliver

Re: IPU - bayer pattern conversion

qiang_li-mpu_se — Thu, 20 Dec 2012 07:29:28 GMT

The two CSI ports have no difference in functions, that should be document issue.

Re: IPU - bayer pattern conversion

yiwulu — Wed, 09 Jan 2013 00:32:49 GMT

Could you please tell me how to configure IPUx_CSIO_CPD_CTRL and IPUx_CSIO_CPD_xxxx for converting bayer format data?

Re: IPU - bayer pattern conversion

qiang_li-mpu_se — Fri, 11 Jan 2013 01:05:18 GMT

Got some comfirmed information from IC designer, for iMX6, the IPU can't support converting bayer format to RGB/YUV on the fly, it can only use generic mode to receive the CSI input bayer data to memory. The CPD can only be used to convert 10 bits color data to 8 bits.

On iMX5x IPU, there is a module called "ISP" which can be used to handle bayer format data, but it was also not fully verified, so in the iMX5x document, the ISP section had been removed. And on iMX6, that "ISP" module had been removed.

Re: IPU - bayer pattern conversion

yiwulu — Wed, 16 Jan 2013 18:06:34 GMT

Thanks for your answer. But I still have another problem with the bayer data. I used the camera test mode to output test data. In theory，the output should be 00 01 02 03 04 05 06 07 ... , but what I captured from the DMA buffer is 00 04 08 0c...

So I lost 3/4 datas. I had checked the waves of the camera output . I am sure that the output is correct. I had checked and test the related settings of the IPU, but no discovery. Can you tell me about what the reason?

A Phenomenon should be noticed. For example: if the camera output 752*480 pixel, the whole data of one frame should be 752*480bytes. Because of data loss ,the whole data must be 752*480/4 . When I used mxc_v4l2_capture to capture one frame data, it thought the data format is yuv420 and whole data is 752*480*1.5 bytes, so the dma buffer size it requests is bigger than the real data. What I got from the buffer like this:

00 04 08....(188byte Sequential datas) 02 06 0A...(another 188 byte Sequential datas) 376 byte random data

04 08 0c....(188byte Sequential datas) 06 0A 0E...(another 188 byte Sequential datas) 376 byte random data (the same random data as above)

then this repeats

I also found that it can configure CPMEM to control the data writing. But it doesn't mention how to configure for Generic data. Is it the problem about CPMEM configuration.

Re: IPU - bayer pattern conversion

qiang_li-mpu_se — Fri, 18 Jan 2013 03:31:13 GMT

To enable the generic mode for CSI, you should pay attention to the followed things:

In CSI_SENS_CONF register, DATA_FORMAT should be set to 0x03 "Bayer or Generic data", PRTCL should be set to 0 "Gated clock mode".

In CPMEM, it is always interleaved mode, PFS "Pixel Format Select" should be 0x6 "Generic data" the total size of frame in CPMEM should be equal to the input frame size. For example, if your input frame size is 752*480*1.5 bytes, due to it is not an interleaved frame, you need adjust the CPMEM setting, you can't set 752 as the width, 12bits bpp 1.5bytes was not supported by generic mode.

In this case, you can config the CPMEM as 282*480*4, the total size is same as 752*480*1.5 bytes, then in CPMEM, all other parameters should be set based on the 282*480 32 bpp frame.

Re: IPU - bayer pattern conversion

yiwulu — Tue, 29 Jan 2013 18:42:51 GMT

After I changed some codes of driver , I can see the raw data image finally. I think Freescale should fix the bugs and add some document about how to deal with raw data

Re: IPU - bayer pattern conversion

fabio_estevam — Wed, 30 Jan 2013 01:54:54 GMT

Could you please share your changes and provide a quick how-to about this topic?

This would help many people.

Regards,

Fabio Estevam

Re: IPU - bayer pattern conversion

Rodrigue — Tue, 26 Feb 2013 09:22:00 GMT

Hi Qiang,

When debayering the CSI input raw image, may I ask you to specify the demosaicing algorithm used in the IPU?

Is it HUE, VNG, bilinear?

I do not see any possibility to specify it.

Re: IPU - bayer pattern conversion

haff — Mon, 25 Mar 2013 17:05:30 GMT

Just to clearify, does this means that IPU processing of bayer data is possible only through system memory? For instance if I want to direct captured video from CSI to IC and apply resize this will not work? Does same goes for crop and rotate?

Regards,

Lavnikevich Dmitry

Re: IPU - bayer pattern conversion

qiang_li-mpu_se — Tue, 26 Mar 2013 01:33:12 GMT

Yes, IPU can't process bayer data (resize, CSC, rotaion), it can only capture them into memory directly in generic data mode.

Re: IPU - bayer pattern conversion

haff — Tue, 26 Mar 2013 06:51:50 GMT

Thank you for reply. Does that includes all non-RGB/YUV formats. For example my camera can work both in bayer and grayscale mode (GREY pixformat). Is this format also cannot be processed directly by IC since it is not RGB/YUV?

Regards,

Lavnikevich Dmitry

Re: IPU - bayer pattern conversion

qiang_li-mpu_se — Tue, 26 Mar 2013 07:28:08 GMT

Yes, the same case for gray.

Re: IPU - bayer pattern conversion

osaka — Tue, 02 Jul 2013 12:02:06 GMT

Mr.Li:

"In CPMEM, it is always interleaved mode, PFS "Pixel Format Select" should be 0x6 "Generic data" the total size of frame in CPMEM should be equal to the input frame size."

in code file /drivers/mxc/ipu3/ipu_param_mem.h, _ipu_ch_param_init(), it looks like that this func do not care about interlace / non-interlace, just set CPMEM value in interlace style. And as u mentioned above, is that means all video should be judged as interlace ?

and,

if 720x480, interlace, CSI-MEM, then, raw data should be 480line( 240 data, 240 blank, 1 data line follow by 1 blank line)

,is it right?

Re: IPU - bayer pattern conversion

qiang_li-mpu_se — Wed, 03 Jul 2013 01:26:16 GMT

Please pay attention, interleave and interlace are not the same thing.

Interleave means the whole video data is put in a whole memory, ipu can access the whole frame with the base memory address.

Non-interleave means the different video data is put in different memory address, for example, Y, U, V data is put in different memory buffer.

Interlace means the video frame is not sent in one time, just like NTSC, for the 720x480 frame, it will be sent on two fields, each field is 720*240 data lines. The whole frame data is 19 blank lines, then 240 data lines, then 3 blank lines; then 20 blank lines, then 240 data lines, then 3 blank lines. But the data in NTSC is interleaved UYVY format.

Re: IPU - bayer pattern conversion

osaka — Wed, 03 Jul 2013 03:57:17 GMT

Mr. Li

yes,u are right, there're many diffs between interlace with progressive ,

it's my mistake, thx