imx6 bayer to yuv conversion

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

imx6 bayer to yuv conversion

10,306 Views
dhananajysharma
Contributor II

!

Hi all,

I have the setup of IMX6 sabre board and APTINA SENSOR MT9P006 5MP.

sensor output is bayer RGB (GRGB format) height & width = 2580X1944

i would like to use ipu for Debayer and save image as YUV420 IPU .

(on the search i found the bayer data can be converted to rgb on fly i tried so many example available on github but cant find any solution)

i tried using opencv its taking about 180 ms wich is too much i want to achieve 15 fps. from a reply i understood ipu cannot be used for debayer use gstreamer.

please suggest some method and example how to convert sensor bayer data to yuv420 using ipu if possible or some other method to achieve 15 fps.

i am cracking my head since month with no result. i have attached my sensor bayer data

Any help will be highly appreciable.

thanks.

rogeriopimentelhacqing_02_AndreSilva

TNSDaianeAngolinii.MX CommunityIPU programming​@

Labels (4)
19 Replies

4,635 Views
aravinthkumarja
Senior Contributor II

Hi All,

Is there any exact procedure to convert camera data from bayer to yuv.?

Regards

Aravinth

0 Kudos

4,636 Views
raymondman
Contributor II

Hi Walter,

Many thanks for your reply. At the first sight of your comments above, I am not aware my problem is related to memory performance.

As the bayer data is come from IPU through DMA where the memory is not cacheable, could you please further elaborate how to use cacheable memory for bayer data?

Many thanks again!

Raymond

0 Kudos

4,636 Views
waltertuppa
Contributor II

Hi,

I wrote my own capture driver (using the IPU) and used the folling code to map the DMA memory (from IPU) to user space: Normally because of DMA memory is mapped uncached, but due to size and usage, it's no problem to map it cached.

/* function to map DMA memory to user space */

static int capture_mmap(struct file *file, struct vm_area_struct *vm)

{

   struct capture_device *dev = file->private_data;

   size_t rsize = vm->vm_end - vm->vm_start;

   int map;

   int idx;

   if ((vm->vm_pgoff * PAGE_SIZE) != (unsigned long)dev->dma)

      return -ENOMEM;           /* wrong base address */

    if (down_interruptible(&dev->busy_lock))

      return -EINTR;

   vm->vm_page_prot = phys_mem_access_prot(file, vm->vm_pgoff, rsize, vm->vm_page_prot);

   /* map DMA memory to user process (allow cacheable, since data much larger than cache) */

   if (remap_pfn_range(vm, vm->vm_start, vm->vm_pgoff, rsize, vm->vm_page_prot))

      map = EAGAIN;

   else

      map = 0;

   up(&dev->busy_lock);

   return map;

}

If you use some driver from FreeScale/NXP, you may have to change the driver.

Regards

Walter

0 Kudos

4,636 Views
raymondman
Contributor II

Hi Walter,

Many thinks for your sharing. It helps me to resolve the problem!

Thanks and regards,

Raymond

0 Kudos

4,636 Views
raymondman
Contributor II

Hi Walter,

I am also doing Bayer data to YUV conversion on an 1280x960 image.

1) constructing bitmap from Bayer data by simply moving memory to construct 4 RGB pixels from a Bayer pixel BGGR

2) use ipu to convert the RGB to YUV

3) use vpu to convert the YUV to JPEG

When I read the Bayer data from a file and do the above processes, the step 1 (Bayer to bitmap) takes ~20ms. But it becomes 146~150ms when it takes data from a MIPI sensor. The MIPI data line shows that the data transfer from the sensor to iMX6 takes 56ms out of 73ms. It seems the step 1 process is blocked at that time and so it needs 2 frame duration to finish the task. I use different cpu core to do the process or divide the task for 2/4 cores but of no use.

Actually I don't believe that the IDMAC is blocking other processes to access memory when it is receiving data from the sensor because you can achieve 2 streams while one of them is HD. But I have no more idea  to improve it.

Can you drop me some hints?

Thanks a lot!

0 Kudos

4,636 Views
waltertuppa
Contributor II

Hi,

time for data from MIPI is much slower, because it is usually stored in non cacheable memory (file data is in cacheable memory).

I had similar timing problems, but using cacheable memory for Bayer Data from MIPI should be no problem, if Bayer Data is larger than  the complete internal Cache (L1/L2), since then old data should be never in cache due to algorithm (we go through all data).

Destination of debayering must be in non cacheable memory or memory must be flushed correctly for further steps with IPU/VPU. Usually writing to non cacheable memory is quite fast (it has not to wait for it), only reading is a problem.

Try cachecable memory for Bayer Data, it's not the normal way, but it should work.

There is already a similar comment from me above.

Regards

Walter

0 Kudos

4,636 Views
RobbieJiang
Contributor IV

Hi Dhananajy,

I'm also working on support for MT9P006 on imx6Q/Linux platform.

But still failed to get the correct image via the V4L2 interface.

Would you please share your driver code with me?

0 Kudos

4,636 Views
dhananajysharma
Contributor II

Hi Walter Tuppa,

Can you please share the timming  of bayer to yuv conversion in CPU single core. What is the sensor MP capacity?

were you getting clear picture because when i do demosac the pictures were very noisy.

0 Kudos

4,636 Views
waltertuppa
Contributor II

Hi,

it was the same sensor as yours, a MT9P006 (5 MP). One  problem with our design was, that the 12bit were wrong aligned in the 16bit data words after stored in memory, so we had to shift them by 4 bits in the NEON assembler code.

We had some problems in the beginning to get the correct data from the CSI interface to memory, but afterwards we got a clear and nice picture (no noise at all). It was quite tricky to get the sensor into right configuration to have 15 fps at full resolution. We wrote a small linux kernel driver to initialize CSI and IPU for our need (using the linux drivers for CSI and IPU from freescale). We didn't use video for linux drivers/code.

Since it was only a prototype, I currently have no hardware on my desk.

We sent two H.264 streams (using the VPU) using RTP over 100MB ethernet. One stream at 1920x1080 (Full HD, 15fps), the other at 640x480 (VGA, 15 fps).

dewarping was done using the GPU (using virtual frame buffers) and YUV 4:4:4 to YUV 4:2:0 conversion was done by the CPU (other core). The video was encoded using the VPU.

4,636 Views
dhananajysharma
Contributor II

Hi Walter,

Thank you for sharing your experience


When i am doing Bayer Data to YUV conversion it is taking around 120 ms which is too much.

i have used NEON optimized method got some bunch of code from github which i added.

but not able to achieve 15 fps.

can you share some information from Bayer to YUV processing code or document method.

i very new to image processing.

Any help will be highly appreciable.

0 Kudos

4,636 Views
waltertuppa
Contributor II

Hi,

the trick was to make the real image smaller. The sensor Bayer is 2580X1944, out RGB/YUV is later only 2576x1940. Our Bayer Matrix is 5x5, so we loose 2 pixel in each border direction. This optimization removes any conditional statements. All internal constants from Bayer conversion are stored in NEON registers.

I used 8x2 super pixels (4 Bayer full pixels) during conversion, that means I convert 8 horizontal pixel in two rows at once from Bayer to RGB (and later YUV). The conversion matrix is fixed for B/Gb and Gr/R for the Bayer Pixel. I also did some optimization in assembler statements to avoid any stalls in pipeline. The conversion only works with NEON registers without any additional memory except the input/output frame buffers.

Other trick is, to say, the Bayer data is cacheable (and store fully aligned). The data is stored there by DMA (so cache does not matter). This leads to no errors, since each frame is much bigger than the cache. But you get a lot more speed, especially, since I access 6 rows (3 Bayer rows) during each conversion (and 4 are always already in cache).

For YUV I also generate for each super pixel a histogram value.

Sorry to say, since this code is owned by my company, I am not allowed to give it to you.

And you are right, the code we found using Google was much slower, since it is much more generalized.

4,636 Views
waltertuppa
Contributor II

Hi,

we had the same problem with this sensor. We finished by using one CPU core to convert BAYER data to RGB using the NEON extension.

Input was 12bit/pixel. (extended to 16bit). RGB was stored 3*8bit/pixel. Alternatively, we directly converted to YUVA with 4*8bit/pixel (easy to use later in OpenCV for dewarping).

CPU usage was about 80 to 90% of one core (at 800MHz), Memory was DDR2L with 266MHz. Due to on error in design, memory was only one bank with 32bit (not two).

If possible, I would insert a FPGA in the datapath to convert Bayer to RGB (uses less power and is more than fast enough.

4,636 Views
chrisroed
Contributor IV

Hi Walter, is there any chance you'd be willing to share the code you used to convert the bayer to rgb?

0 Kudos

4,636 Views
dhananajysharma
Contributor II

thanks for reply rogerio,

from the datasheet i understand IPU cannot able to perform bayer to YUV conversion.


Can we use Display Processor (DP) in imx6 which performs all the processing required for data sent to a display.

if i sent the bayer Input from system memory to DP.
can it convert bayer data to yuv as per datasheet. will this work ?

0 Kudos

4,635 Views
andre_silva
NXP Employee
NXP Employee

Just complementing a bit the answer, if you are going to use the GPU, you will need to feed the correct packed data to be used as invalid input for the shader (correct bbp), which may mean loosing some performance by pre-processing it with CPU. Unless you already have the correct input format.

How is the Bayer RGB are you using ?

cheers,

Andre

0 Kudos

4,636 Views
andre_silva
NXP Employee
NXP Employee

** sorry, I meant VALID input for shader !

0 Kudos

4,636 Views
rogerio_silva
NXP Employee
NXP Employee

No. IPU (IC, DP, DC and etc) doesn't recognize bayer as a valid image format.

It can receive bayer format from CSI port as generic data.

4,636 Views
rogerio_silva
NXP Employee
NXP Employee

Hi dhananajysharma​,

i.MX53 and i.MX6 IPU don't make Bayer conversion.

You will need to use some graphics library/API like gstreamer, ffmpeg and others to make this conversion using the CPU.

4,636 Views
benhenricksen
Contributor III

What I don't understand is why the manual talks about handling Bayer data if it is not supported?! There are 27 references to it.

There is even a error interrupts for IC_BAYER_BUF_OVF & IC_BAYER_FRM_LOST_ERR

9.2.1.1 Data formats supported include Raw (Bayer),

The first item in the Table 37-2. Data Formats Supported By The Camera Port is Bayer RGB

0 Kudos