IPU - bayer pattern conversion

[tollerhecht]

Hi,

I'm planning to use a i.mx6 in an digital imaging application. The intention is to transfer image data from a FPGA device to the i.mx6 memory using the parallel CSI ports. The format of the image data is a bayer pattern. In the documention I can see that the IPU can receive bayer pattern and generic image data. I'm not shure how the IPU really handles the bayer data and what's the difference. Does the IPU unit contain any bayer demosaicing hardware ? Is the bayer data automatically converted to RGB or YUV when transfering to memory ?

Can anybody help me ?

Thanks and best regards

Oliver

[ivankozic]

No actually, MIPI is only serialized parallel protocol - everything is embedded in it (FV, LV and DV and data) - similar to HiSPi from Aptina. MIPI receiver on i.MX6 only deserialzes this serial stream and reassembles the original parallel stream. MIPI is only there to provide higher speeds, less conductors and longer bus length. So it matters a lot how FV, LV and DV are generated by Aptina and what IPU expects (for some reason it expects something more like HSYNC and VSYNC than xV signals). Incidentally, Aptina's support is terrible as well (we are using AR0330), so combining i.MX6 with Aptina seems like a bad combo :smileyhappy: Aptina's SW department (drivers@aptina.com) never actually tested MIPI for AR0330 - you should see the pseudo-code file they've sent me and their comments. AR0330 is also quite buggy on hardware level.

Anyway, due to MIPI receiver limitations, we have moved to using parallel interface on IPU, and I'm playing around with this now - seems to be much more flexible than MIPI.

Regarding your question - it seems rather difficult to make IDMAC work as specified when using something other than 8-bit - as I said before - even if I ignore the fact that I cannot sync with MIPI, I could not get the byte ordering to work properly with it (12bit). Ignore that I am talking about MIPI - when talking about IDMAC and CPMEM, MIPI is already deserialized and practically same like parallel interface. There is one big difference - I think that maybe it's more flexible with parallel in this case, as you can maybe configure CPMEM correctly with parallel - with MIPI it's impossible, since SENS_DATA_FORMAT and DATA_WIDTH registers are ignored and this info is acquired through MCT_DI bus.

I also saw the other thread you've made - I'll try to see it in more detail later, since now I have no time, but to speed things up, you can tell me maybe which width are you using for CPMEM? Are you using 12-bit or 16-bit width?

[DraganOstojic]

I found the fix for the corruption today. How I found it is another story, but it turns out, for 16-bit generic capture to work you need to set IDMAC CPMEM to the following (in addition to other changes I listed in another thread):

In _ipu_ch_param_init():

  case IPU_PIX_FMT_GENERIC_16:

  /*Represents 16-bit Generic data */

  ipu_ch_param_set_field(&params, 0, 107, 3, 3);  /* bits/pixel */

  ipu_ch_param_set_field(&params, 1, 85, 4, 6);   /* pix format */

  ipu_ch_param_set_field(&params, 1, 78, 7, 7);   /* burst size */

  break;

This is unexpected as RM is very specific that burst size should be set to 32 for 16BPP.

[ivankozic]

Hmm, interesting discovery - I have also tried several values here, but it did not work for me. RM is not that restrictive for it though - for 16bpp you can set any value from 1 to 32 pixels for burst size.

Of course, downside is that more overhead is generated with 8 bits burst length thus lowering the effective bandwidth to the memory.

Have you tried with any other value greater than 8 and lower than 32? In order to maximize the bandwidth, I would try all the values from 9-31 :smileyhappy:

[DraganOstojic]

Thanks for the clarification. I checked the range you suggested and anything above 8 doesn't work so this is already at a maximum. I'm capturing 2592x1944x16 bits@7fps/48Mhz pixel clock. I'm puzzled that the "working" value for 2x transfer rate is at 1/4 of the allowable range, compared to 8-bit mode same resolution and frame rate which works at the max of the range 64.

[ivankozic]

Bummer, 7fps are not really usable... I guess overhead is too big for any usable frame-rate. Don't be puzzled - judging from what I've seen in drivers so far, Freescale doesn't really take care much of them - most of the drivers are written with only basic functionality implemented, often having major differences to what RM states. Everything else is left up to us. I am actually puzzled when I see a driver work as specified from stock :smileyhappy:

So it might be something completely stupid that you're overlooking (99% of the time it is, at least for me), but it'll take days to find out what, I know - been there...

[DraganOstojic]

On this system we could go as high as 15fps, my processing takes ~20ms/frame in the GPU but with this pixel clock (48 MHz) Aptina camera outputs frames at 7fps. We decided not to use internal camera pll because of some "striation" artifacts observed when PLL is used.

[DraganOstojic]

I'm using 16-bit width when programming CPMEM:

PFS 0x6, BPP 0x3, NPB 0x1f

I'm trying to get support from Freescale through managerial channels and I'll keep you in the loop.

[joštnovak]

I managed to get 10bit bayer data from mipi camera as 16bit data in memory.

I have set CPMEM:

     16bit/pixel, generic format and 32bytes burst.

It's true, that height/width/stride are passed by V4L, but you should take a look at format settings in  driver/media/video/mxc/capture/mxc_v4l_capture.c , i added following lines:

case V4L2_PIX_FMT_SBGGR10:

     size = f->fmt.pix.width * f->fmt.pix.height * 2;

     bytesperline = f->fmt.pix.width * 2;

     break;

Values in memory get "expanded" to 16 bits. For example

10-bit value 0x3FF is written as 0xFFFF

10-bit value 0x000 is written as 0x0000

I'm not sure about other values, since my camera only outputs test values 0x000 and 0x3FF.

Regards,

Novak Jost

[kennywang]

Dear Jost

You said the Byare 10 bit values in  in memory get "expanded" to 16 bits. For example

10-bit value 0x3FF is written as 0xFFFF

10-bit value 0x000 is written as 0x0000

In memory I receive 0x0A 0x28  data how can I conver the data to 10-bit value ?

[davemcmordie]

Jošt,

How did you manage to get the 10 bit generic capture working?  I am trying to get data from an OV5647, and everything works except the frame interrupt.  Following the advice of this and another thread, I've set my BPP to 0x3 (16 bit), NPB to 0x1f (32 pixels/burst) adn PFS to 0x6 (Bayer).  Scope shows clock and data, MIPI status 0x300/0x330 (mipi error clear).  Have double and triple checked that I am on IPU0 CSI0 with MIPI interface enabled (GPIO 19 correctly).  Sensor is using virtual channel 0.

Following your lead I have added support for 10 bit Bayer/Generic throughout the code and selected this format in my driver.

Was there a point where you struggled with the lack of the frame interrupt and something comes to mind?

One thing that is bugging me is that I don't understand the difference between the "CSI IC MEM" input and the "CSI MEM" input.  I guess I can't use the Image Converter block (IC) with 10 bit bayer, but when I select the CSI MEM input I get the same result.

Dave

[joštnovak]

Your settings seem OK, MIPI status registers look fine as well...

Do you get any data in memory at all?

CSI->MEM is direct path from CSI to memory. CSI->IC->MEM path goes through Image converter (for example for image resizing). Image converter does not support bayer format only specific formats like RGB and YUV...

[davemcmordie]

I finally got it working a couple of days ago.  The IC can definitely not be in the loop.  I had a couple of problems but basically I hit and resolved the following issues:

1. MIPI from the camera -- I had the wrong register settings-- I was missing the setting that told the sensor to start streaming MIPI data.
2. Eliminating CSI->IC->MEM path in mxc_v4l2_capture-- I just took out the code that made this an option.
3. Switching to 8 bit sensor output.  Shouldn't have to do this, but I was unable to get the image size right with 10 bit data (I was using 2 * width * height).  This seems to have been a big part of the missing EOF interrupt.
4. Using Bayer instead of Grey.  Not sure why, but I couldn't get Bayer working.  When I added support for Grey, I started getting images.  Bayer should work, but doesn't for me.
5. Made the capture client request Grey images.  If the capture client was requesting the image in a different format (default was UYVY or something), then the image would appear with the wrong format.

At this point I am getting Bayer patter data in memory and will soon look into using the graphics coprocessor to handle demosaicing.

Thanks for your posts in this thread.  Together with Dragan and Ambiguity's posts they led me to a solution.

[ivankozic]

The RM specifies what you're saying in CSI chapter of IPU (37.4.3.7 Packing to Memory). It states there that 9-16 bit data are automatically extended to 16 bits if using regular packing. Which would be great.

So after this, I have tried initializing everything to 16 bits (including mxc_v4l2_capture.c - I've put bytesperline and stride just like you did - before it was X*3/2 for 12bits), except for the MIPI info structure (I have set MIPI_DT_RAW12 there). This all should somehow be sufficient to receive 12bit data to 16bit mem locations.

However, nothing happens - the data is still packed 12bit (and it seems that it's packed long before it gets to IDMAC - I think CSI port is doing the packing). I thought I was missing something, but then I read the following in the RM (37.4.3.2.2 Chapter) - In MIPI mode, SENS_DATA_FORMAT and DATA_WIDTH are ignored and fetched from MIPI receiver via MCT_DI bus (which means it gets 12bit width from MIPI).

It's still not impossible that I have missed something - I'll try to debug and check, but I think it's not so easy for MIPI to pack to 12bit data to 16bit locations. It also seems weird that your data is "expanded" as you say to 16bits by adding '1's instead of zeros. As far as I could see, only compander unit can do such "expanding" - IDMAC via CPMEM should only fill the unused bits with '0's. Also, compander should be off for MIPI RAW data reception, as it isn't possible to process any Generic/Bayer/raw data with compander. In fact it can only pack 10bit data to 8bit, as Mr. Li pointed out.

[qiang_li-mpu_se]

There are some discussion for 16 bits generic data capture in https://community.freescale.com/thread/308634

I think it can also be appied on your case, although your input is 12 bits.

[ivankozic]

Ok, I've read it again - not really sure why is there a distinction between Generic data and Bayer data, as IPU driver treats it the same. Anyway, it seems that it is possible to do this, but the 12bit pixel gets written into MSB of a 16bit word, which is still fine for me. However, I have read on several other threads that IPU and MIPI need to have a synchronized format, but I have never tried it, so I can't confirm.

I'll try to reconfigure everything to 800x600x2 except for MIPI regs, which will stay 12bit. So to summarize for someone who maybe follows this:

1. V4L config - mxc_v4l2_capture.c : reconfigure mxc_v4l2_s_fmt() function for correct bytesperline and size.

2. CPMEM config - ipu_param_mem.h : reconfigure _ipu_ch_param_init() function to work with IPU_PIX_FMT_GENERIC in 16bit mode. Stride and width/height come from V4L and should be the same, no recalc necessary (as for 12bit pack), but BPP and DMA burst size need to change.

3. For MIPI, as far as I can see, we only need to do mipi_csi2_set_datatype(mipi_csi2_info, MIPI_DT_RAW12) inside of the actual sensor driver (which is already done in my case).

I'll post back the results.

[osaka]

Mr.Li:

"In CPMEM, it is always interleaved mode, PFS "Pixel Format Select" should be 0x6 "Generic data" the total size of frame in CPMEM should be equal to the input frame size."

in code file /drivers/mxc/ipu3/ipu_param_mem.h,  _ipu_ch_param_init(),  it looks like that this func do not care about interlace / non-interlace, just set CPMEM value in interlace style. And as u mentioned above, is that means all video should be judged as interlace ?

and,

if 720x480, interlace,   CSI-MEM,  then, raw data should be 480line( 240 data, 240 blank,  1 data line follow by 1 blank line)

,is it right?

[qiang_li-mpu_se]

Please pay attention, interleave and interlace are not the same thing.

Interleave means the whole video data is put in a whole memory, ipu can access the whole frame with the base memory address.

Non-interleave means the different video data is put in different memory address, for example, Y, U, V data is put in different memory buffer.

Interlace means the video frame is not sent in one time, just like NTSC, for the 720x480 frame, it will be sent on two fields, each field is 720*240 data lines. The whole frame data is 19 blank lines, then 240 data lines, then 3 blank lines; then 20 blank lines, then 240 data lines, then 3 blank lines. But the data in NTSC is interleaved UYVY format.

[osaka]

Mr. Li

yes,u are right, there're many diffs between interlace with progressive ,

it's my mistake, thx

[yiwulu]

After I changed some codes of driver , I can see the raw data image finally. I think Freescale should fix the bugs and add some document about how to deal with raw data

[fabio_estevam]

Could you please share your changes and provide a quick how-to about this topic?

This would help many people.

Regards,

Fabio Estevam

[tollerhecht]

Qiang,

Thank you very much for your answer, this was very helpful.

For the CSI-1 port the IPUx_CSI1_CPD_CTRL register is filled with reserved bits. Does this work only on the CSI-0 port ? Another problem is, that the images I want to convert have a header that contain some special information (timestamp, etc.). Is there a way to prevent the header data from being modified, when using the bayer conversion (skip) ?

Thanks and best regards,

Oliver