IPU - bayer pattern conversion

[tollerhecht]

Hi,

I'm planning to use a i.mx6 in an digital imaging application. The intention is to transfer image data from a FPGA device to the i.mx6 memory using the parallel CSI ports. The format of the image data is a bayer pattern. In the documention I can see that the IPU can receive bayer pattern and generic image data. I'm not shure how the IPU really handles the bayer data and what's the difference. Does the IPU unit contain any bayer demosaicing hardware ? Is the bayer data automatically converted to RGB or YUV when transfering to memory ?

Can anybody help me ?

Thanks and best regards

Oliver

[yaolinchang]

Hi All,

       (1)Does anyone know how we set camera path from "CSI-IPU-MEM" to "CSI-MEM" ?

       (2)Where is the memory that save image buffer after capture?

       (3)We want to get 12 bit raw data from CSI to MEM directly, and skip IPU process.

            How should we do? Which functions are we need to modify?

Thanks,

Best Regards,

Yao

[twdebugguy]

Please make sure the IPU_CSI0_MUX and IPU_CSI1_MUX correct.

IPU2 parallel port on IMX6Q

[qiang_li-mpu_se]

    The CSI module can support the bayer format, you need enable the CSI CPD function in register IPUx_CSIO_CPD_CTRL, and after that, you also need config the IPUx_CSIO_CPD_xxxx registers based on the bayer detail format, GBGB or GRGR.

    If you want to convert the bayer data to YUV/RGB from CSI  to system memory, then CSI can only handle the 8bits MSB for each R/G/B in bayer format. Just like some other YUV camera sensors, you only need add the CSI CPD related config to support the bayer image, all other codes can be re-used in this mode.

    There is another method to capture the bayer image from CSI, it is the generic mode, in this mode, the bayer image can be captured into memory as it's raw data, if it is 12bits bayer image, each 12 bits R/G/B will be captured into memory in each 16bits (Only 12 bits MSB has the valid data), in this mode, the in memory bayer data can't be handled by IPU directly, only CSI module can handle the bayer data directly. You need use software to convert that bayer image to other format.

[yiwulu]

Could you please tell me how to configure IPUx_CSIO_CPD_CTRL and  IPUx_CSIO_CPD_xxxx for converting bayer format  data?

[qiang_li-mpu_se · Accepted solution]

    Got some comfirmed information from IC designer, for iMX6, the IPU can't support converting bayer format to RGB/YUV on the fly, it can only use generic mode to receive the CSI input bayer data to memory. The CPD can only be used to convert 10 bits color data to 8 bits.

    On iMX5x IPU, there is a module called "ISP" which can be used to handle bayer format data, but it was also not fully verified, so in the iMX5x document, the ISP section had been removed. And on iMX6, that "ISP" module had been removed.

[haff]

Just to clearify, does this means that IPU processing of bayer data is possible only through system memory? For instance if I want to direct captured video from CSI to IC and apply resize this will not work? Does same goes for crop and rotate?

Regards,

Lavnikevich Dmitry

[qiang_li-mpu_se]

Yes, IPU can't process bayer data (resize, CSC, rotaion), it can only capture them into memory directly in generic data mode.

[haff]

Thank you for reply. Does that includes all non-RGB/YUV formats. For example my camera can work both in bayer and grayscale mode (GREY pixformat). Is this format also cannot be processed directly by IC since it is not RGB/YUV?

Regards,

Lavnikevich Dmitry

[qiang_li-mpu_se]

Yes, the same case for gray.

[davidsoto]

Hi,

   I am also trying to capture on gray scale using 8 bits per pixel. I need to generate the signal using an FPGA but I was not sure about the timing. It seems that:
1) I should use discrete sync signals
2) I should connect the data on pins D[19:4] but since I would use 8 bits per pixel it should be in D[19:12]

3) I should use a pixel clock of:

1920x1080x30x1.35x1

In order to get 30fps.

Does this sound right?

Maybe I could check the datasheet of one of the cameras that you used for this for a better reference, could you provide one example camera number?

Thanks,

-David

[yiwulu]

Thanks for your answer. But I still have another problem with the bayer data. I used the camera test mode to output test data. In theory，the output should be 00 01 02 03 04 05 06 07 ... , but what I captured from the DMA buffer is 00 04 08 0c...

So I lost 3/4 datas. I had checked the waves of the camera output . I am sure that the output is correct.  I had checked and test  the related settings of the IPU,  but no discovery. Can you tell me about what the reason?

A Phenomenon should be noticed. For example:  if the camera output 752*480 pixel, the whole data of one frame should be 752*480bytes. Because of data loss ,the whole data must be 752*480/4 .  When I used mxc_v4l2_capture  to capture  one frame data, it thought the data format is yuv420 and whole data is 752*480*1.5 bytes, so the dma buffer size it requests is bigger than the real data. What  I got from the buffer like this:

00 04 08....(188byte Sequential datas) 02 06 0A...(another 188 byte Sequential datas) 376 byte random data

04 08  0c....(188byte Sequential datas) 06 0A 0E...(another 188 byte Sequential datas) 376 byte random data (the same random data as above)

then this repeats

I also found that it can configure CPMEM to control the data writing. But it doesn't mention how to configure for Generic data. Is it the problem about CPMEM configuration.

[qiang_li-mpu_se]

To enable the generic mode for CSI, you should pay attention to the followed things:

In CSI_SENS_CONF register, DATA_FORMAT should be set to 0x03 "Bayer or Generic data", PRTCL should be set to 0 "Gated clock mode".

In CPMEM, it is always interleaved mode, PFS "Pixel Format Select" should be 0x6 "Generic data" the total size of frame in CPMEM should be equal to the input frame size. For example, if your input frame size is 752*480*1.5 bytes, due to it is not an interleaved frame, you need adjust the CPMEM setting, you can't set 752 as the width, 12bits bpp 1.5bytes was not supported by generic mode.

In this case, you can config the CPMEM as 282*480*4, the total size is same as 752*480*1.5 bytes, then in CPMEM, all other parameters should be set based on the 282*480 32 bpp frame.

[ivankozic]

Hi Mr. Li,

One short question - I am currently trying out something similar as in the original question, but with using MIPI, instead of parallel. I am trying to get a Bayer SVGA frame (800x600), 12bits per pixel.

As far as I can see from RM, I should use NON_GATED clock mode, since it is MIPI and according to your writing, I should configure CPMEM as 300x600x4 interleaved, instead of 800x600x1.5 interleaved as would be the actual case.

I configured all the other IPU registers for MIPI reception of 800x600x1.5 frame.

Could you confirm this?

[qiang_li-mpu_se]

I think there should be some setting in MIPI module to map the 12bits input, but I had nerver used MIPI camera, so I can't confirm it.

[ivankozic]

Thanks for a quick reply. The MIPI supports 12bit output, but I thought that IPU has some limitation - to quote you:

"due to it is not an interleaved frame, you need adjust the CPMEM setting, you can't set 752 as the width, 12bits bpp 1.5bytes was not supported by generic mode."

I am not sure where this limitation is, since I saw that CPMEM could be configured as 12bit as well. Do you know why 12bits bpp generic mode is not supported?

I have configured everything except CPMEM to 800x600x1.5. CPMEM is configured as 300x600x4 and it all seems fine. The only trouble I have is that it would be more convenient to pack the frame to the memory as 16bit words with the highest nibble nulled than current 12bit packing. But I can't seem to find out how this is done - I guess IDMAC could do this if configured as 12-bits (I am pretty sure CPMEM could be configured this way), but that brings up the original question - Can I at all use the 12bpp Generic mode setting in CPMEM?

[qiang_li-mpu_se]

I think you should set 16bpp mode in CPMEM and set 12 bit data width in CSI register. And in CPMEM, the total data size for one frame should be 800*600*2. For the CSI to memory use case, the data can only be packed as 8/16/32 bits, it doesn't support data packed to12bits.

You can reference to "Table 37-2. Data Formats Supported By The Camera Port" iMX6 reference manual.

[ivankozic]

So far I haven't succeeded - I get garbage all the time and it is not really in sync with RM. I have tried both 16bit and 12bit configurations, I'll do some more trying, but I guess this is not really possible. 12bit is completely unusable, so I won't mention that case.

For 16bit case:

There is a nice equation on page 2729 of RM (Rev1), Chapter 37.4.2.7:

ADDR = EBA + (XB + SX)*BPP + (YB + SY)*(SL + 1)

However, system behaves differently. According to this, Pixel (x = 0, y = 0) should be placed at EBA, while Pixel (x = 1, y = 0) should be placed at EBA + 1*BPP (EBA + 16bit in case of 16bpp). However, this is not the case as for the first frame I get 32 bytes of packed data, and then 32 bytes of garbage data (burst size is 32 pixels). Afterwards I get interleaved lines. To simplify:
A = 32 bytes of valid data,

G = 32 bytes of garbage,

X = 32 bytes of what seems to be valid data, but isn't (lines from the previous frame or similar).

Memory looks like:

1. frame: AGAGAGAGAGAGAGAG,

2..n frames: AXAXAXAXAXAXAXAX,

This all shows that DMA is bursting 32 bytes of packed 12-bit data, but for the next line it bursts whatever is in IPU memory back to system memory - this is why for the first frame it bursts garbage, and for the next frames it bursts the leftovers from the previous frame(s). This is all when first started after boot. When started for second, third or n-th time it always shows AXAXAX patterns (no garbage per say), which makes sense, as IPU memory is already full of 'valid' data from the previous application run.

For the case with 16bit data and 1 pixel burst size, it gets really weird - I get the 130 bytes from RG line and then 130 bytes from GB line (?!?) - I have no idea what is going on in this case - it could be that IPU->System memory bandwidth is too low and it's somehow missing pixels, but I could easily be wrong.

Ok, so if you have the nerves to read this essay :smileyhappy:, have you got any further suggestions on how to put 12bit packed data to 16bit memory?

[DraganOstojic]

Ivan, did you resolve this issue with garbled data? What did you do? I'm at the similar point with my effort but still getting bad data in memory.

[ivankozic]

No I haven't really - at some point it got better, but then I realized that I couldn't get syncing. In fact I used all of this for Aptina sensor which uses FV instead of VSYNC so my stream was always missing the frame boundary. In general, maybe this could have been fixed somehow (but I have tried with inverting VSYNC in the driver code), but for now we have moved away from MIPI to parallel interface.

And also by "got better" I don't really mean much better - 12bit mode works quite weird. In fact I think the biggest culprit is this MIPI_DI bus which automatically provides MIPI info to IPU. This way there's really not much room for playing around.

[DraganOstojic]

Ivan, from what you are saying about VSYNC and FV it appears that you were working with parallel interface but then you say you were moving away from mipi to parallel. Can you please confirm that you are seeing issue with parallel?

I have issue with 12-bit on parallel interface and Aptina at9p031 sensor.

I posted more details in the following thread:

16-bit generic capture not working

I'm out of ideas what could I do next with parallel interface so I'll try to see if 16-bit raw can be made working when input is from the mipi camera.