I have the setup of IMX6 sabre board and APTINA SENSOR MT9P006 5MP.
sensor output is bayer RGB (GRGB format) height & width = 2580X1944
i would like to use ipu for Debayer and save image as YUV420 IPU .
(on the search i found the bayer data can be converted to rgb on fly i tried so many example available on github but cant find any solution)
i tried using opencv its taking about 180 ms wich is too much i want to achieve 15 fps. from a reply i understood ipu cannot be used for debayer use gstreamer.
please suggest some method and example how to convert sensor bayer data to yuv420 using ipu if possible or some other method to achieve 15 fps.
i am cracking my head since month with no result. i have attached my sensor bayer data
Any help will be highly appreciable.
Many thanks for your reply. At the first sight of your comments above, I am not aware my problem is related to memory performance.
As the bayer data is come from IPU through DMA where the memory is not cacheable, could you please further elaborate how to use cacheable memory for bayer data?
Many thanks again!
I wrote my own capture driver (using the IPU) and used the folling code to map the DMA memory (from IPU) to user space: Normally because of DMA memory is mapped uncached, but due to size and usage, it's no problem to map it cached.
/* function to map DMA memory to user space */
static int capture_mmap(struct file *file, struct vm_area_struct *vm)
struct capture_device *dev = file->private_data;
size_t rsize = vm->vm_end - vm->vm_start;
if ((vm->vm_pgoff * PAGE_SIZE) != (unsigned long)dev->dma)
return -ENOMEM; /* wrong base address */
vm->vm_page_prot = phys_mem_access_prot(file, vm->vm_pgoff, rsize, vm->vm_page_prot);
/* map DMA memory to user process (allow cacheable, since data much larger than cache) */
if (remap_pfn_range(vm, vm->vm_start, vm->vm_pgoff, rsize, vm->vm_page_prot))
map = EAGAIN;
map = 0;
If you use some driver from FreeScale/NXP, you may have to change the driver.
I am also doing Bayer data to YUV conversion on an 1280x960 image.
1) constructing bitmap from Bayer data by simply moving memory to construct 4 RGB pixels from a Bayer pixel BGGR
2) use ipu to convert the RGB to YUV
3) use vpu to convert the YUV to JPEG
When I read the Bayer data from a file and do the above processes, the step 1 (Bayer to bitmap) takes ~20ms. But it becomes 146~150ms when it takes data from a MIPI sensor. The MIPI data line shows that the data transfer from the sensor to iMX6 takes 56ms out of 73ms. It seems the step 1 process is blocked at that time and so it needs 2 frame duration to finish the task. I use different cpu core to do the process or divide the task for 2/4 cores but of no use.
Actually I don't believe that the IDMAC is blocking other processes to access memory when it is receiving data from the sensor because you can achieve 2 streams while one of them is HD. But I have no more idea to improve it.
Can you drop me some hints?
Thanks a lot!
time for data from MIPI is much slower, because it is usually stored in non cacheable memory (file data is in cacheable memory).
I had similar timing problems, but using cacheable memory for Bayer Data from MIPI should be no problem, if Bayer Data is larger than the complete internal Cache (L1/L2), since then old data should be never in cache due to algorithm (we go through all data).
Destination of debayering must be in non cacheable memory or memory must be flushed correctly for further steps with IPU/VPU. Usually writing to non cacheable memory is quite fast (it has not to wait for it), only reading is a problem.
Try cachecable memory for Bayer Data, it's not the normal way, but it should work.
There is already a similar comment from me above.
I'm also working on support for MT9P006 on imx6Q/Linux platform.
But still failed to get the correct image via the V4L2 interface.
Would you please share your driver code with me?
Hi Walter Tuppa,
Can you please share the timming of bayer to yuv conversion in CPU single core. What is the sensor MP capacity?
were you getting clear picture because when i do demosac the pictures were very noisy.
it was the same sensor as yours, a MT9P006 (5 MP). One problem with our design was, that the 12bit were wrong aligned in the 16bit data words after stored in memory, so we had to shift them by 4 bits in the NEON assembler code.
We had some problems in the beginning to get the correct data from the CSI interface to memory, but afterwards we got a clear and nice picture (no noise at all). It was quite tricky to get the sensor into right configuration to have 15 fps at full resolution. We wrote a small linux kernel driver to initialize CSI and IPU for our need (using the linux drivers for CSI and IPU from freescale). We didn't use video for linux drivers/code.
Since it was only a prototype, I currently have no hardware on my desk.
We sent two H.264 streams (using the VPU) using RTP over 100MB ethernet. One stream at 1920x1080 (Full HD, 15fps), the other at 640x480 (VGA, 15 fps).
dewarping was done using the GPU (using virtual frame buffers) and YUV 4:4:4 to YUV 4:2:0 conversion was done by the CPU (other core). The video was encoded using the VPU.
Thank you for sharing your experience
When i am doing Bayer Data to YUV conversion it is taking around 120 ms which is too much.
i have used NEON optimized method got some bunch of code from github which i added.
but not able to achieve 15 fps.
can you share some information from Bayer to YUV processing code or document method.
i very new to image processing.
Any help will be highly appreciable.
the trick was to make the real image smaller. The sensor Bayer is 2580X1944, out RGB/YUV is later only 2576x1940. Our Bayer Matrix is 5x5, so we loose 2 pixel in each border direction. This optimization removes any conditional statements. All internal constants from Bayer conversion are stored in NEON registers.
I used 8x2 super pixels (4 Bayer full pixels) during conversion, that means I convert 8 horizontal pixel in two rows at once from Bayer to RGB (and later YUV). The conversion matrix is fixed for B/Gb and Gr/R for the Bayer Pixel. I also did some optimization in assembler statements to avoid any stalls in pipeline. The conversion only works with NEON registers without any additional memory except the input/output frame buffers.
Other trick is, to say, the Bayer data is cacheable (and store fully aligned). The data is stored there by DMA (so cache does not matter). This leads to no errors, since each frame is much bigger than the cache. But you get a lot more speed, especially, since I access 6 rows (3 Bayer rows) during each conversion (and 4 are always already in cache).
For YUV I also generate for each super pixel a histogram value.
Sorry to say, since this code is owned by my company, I am not allowed to give it to you.
And you are right, the code we found using Google was much slower, since it is much more generalized.
we had the same problem with this sensor. We finished by using one CPU core to convert BAYER data to RGB using the NEON extension.
Input was 12bit/pixel. (extended to 16bit). RGB was stored 3*8bit/pixel. Alternatively, we directly converted to YUVA with 4*8bit/pixel (easy to use later in OpenCV for dewarping).
CPU usage was about 80 to 90% of one core (at 800MHz), Memory was DDR2L with 266MHz. Due to on error in design, memory was only one bank with 32bit (not two).
If possible, I would insert a FPGA in the datapath to convert Bayer to RGB (uses less power and is more than fast enough.
thanks for reply rogerio,
from the datasheet i understand IPU cannot able to perform bayer to YUV conversion.
Can we use Display Processor (DP) in imx6 which performs all the processing required for data sent to a display.
if i sent the bayer Input from system memory to DP.
can it convert bayer data to yuv as per datasheet. will this work ?
Just complementing a bit the answer, if you are going to use the GPU, you will need to feed the correct packed data to be used as invalid input for the shader (correct bbp), which may mean loosing some performance by pre-processing it with CPU. Unless you already have the correct input format.
How is the Bayer RGB are you using ?
What I don't understand is why the manual talks about handling Bayer data if it is not supported?! There are 27 references to it.
There is even a error interrupts for IC_BAYER_BUF_OVF & IC_BAYER_FRM_LOST_ERR
184.108.40.206 Data formats supported include Raw (Bayer),
The first item in the Table 37-2. Data Formats Supported By The Camera Port is Bayer RGB