i.MX6 IPU and rotating a frame buffer

[jasonhaedt]

I'll try to make a long story as short as possible :smileyhappy: (let me know if I skip any important details).  Hopefully this is an easy one, but we're still a little green on this part and the IPU is a pretty complex beast, so I'm banging my head at the moment.  The LCD we're planning on using for this product is FHD/portrait mode (so, 1080x1920) connected via DSI.  We've written a driver for this to glue in to the existing DSI driver (so, pretty much like the hx8369 driver down in drivers/video/mxc).  So far so good - everything works at this point.  However, the physical orientation of the display will be landscape i.e. 1920x1080, so we were hoping to use the IPU's IRT to rotate the final framebuffer 90 degrees.  In order to not have to write plugins for our userspace application(s) to worry about setting up the IPU to do this (and also just to try to get a proof-of-concept going), I've tried to modify the IPU framebuffer driver (mxc_ipuv3_fb.c) to accommodate this.  The quick hack to test this out was just to have the driver allocate an extra framebuffer down in mxcfb_map_video_memory (we'll call this new buffer A and the original buffer B).  This new buffer is the one that now gets attached to the kernel framebuffer i.e. fb_info->screen_base (so this is the one that will get rendered in to).  I set up DMA through the IRT using MEM_ROT_VF_MEM (channel 46 in, channel 49 out) with buffer A being the input and buffer B being the output (initializing both with ipu_init_channel_buffer) (the rotation parameter is set to IPU_ROTATE_90_RIGHT, input resolution is x=1920/y=1080/stride=(3*1920) (we're running rgb/24bpp), output resolution is x=1080/y=1920/stride=(3*1080)).  Finally I link the rotation channel to the display channel (ipu_link_channels, MEM_ROT_VF_MEM -> MEM_FG_SYNC).

With this setup, my rotated screen is pretty much garbage (so much so that there's no discernible pattern to guess as to what might be wrong).  The weird part though is if I cut down horizontal resolution of the input framebuffer (and of course the framebuffer structure that gets registered) to 1080x1080, the rotation works fine (I believe I've tried non-square resolutions too with a lower x resolution and they worked okay, but I'll have to double-check that one).

Okay, I just did some more digging in to the underlying registers that are getting configured by the IPU driver and I see that the width/height fields in IPUx_IC_IDMAC_3/4 are limited to 10 bits (1024 pixels), so I assume that I'd need to figure a way to break the 1080x1920 rotation in to four separate DMA transfers, correct?

[rizwanm]

Hi ,

Anyone have PATCH for 180 degree rotation for Linux Kernel 4.15 .

Thanks in advance

Best Regards,

Rizwan M

[luwinbo]

Does anyone has this patch for bsp 4.1.15?

[markusbraitner]

The patch from Frieder Schrempf has helped me out to be able to rotate the display on the 3.10.17 NXP kernel (=> many thanks!!!). However, we're now trying to migrating to 4.1.15 NXP kernel. Has anybody maybe already ported the patch to this kernel version or has some other solution?

Kind regards,

Markus

[alexeykuzmin]

Hi, Markus.

We are using NXP kernel version 4.9 and also looking solution for frame-buffer rotation issue because it still exist.

Did your found something for ver 4.1 finally?

[jemish_1990]

Hi All,

Based on patches provide by Jerome and Cassol I am uploading one patch more patch for framebuffer  rotation.I am not expert of IPU but  I have done some additional work on top that.
I have uploaded that patch here for reference. It is created on kernel 3.14.28.

This patch has following additional fixes/changes:

• It fixes below errors:
     mxc_sdc_fb fb.17: Error updating SDC buf 1 to address=0x42F00000, current buf 2, buf0 ready 1, buf1 ready 0, buf2 ready 0.
• Take care of parameters checking for overlay layer. Original patch is limited to BG layer of primary display it didn't take care parameter
     checking for overlay layer(fb1) of primary display.
• With original patch you can't run preview usecase (capture->display) due to MEM_ROT_VF_MEM channel. Because in preview (capture->display) use case IPU is checking few channels are busy or not. So I have setup DMA through IRT using different channel (MEM_ROT_ENC_MEM). In short this patch will be help in usecase where user want to run GUI on one layer (fb0 or BG) and want to display video on overlay layer (fb1 or FG) using alpha blending simultaneously. Please note that to rotate (overlay)layer user need to pass rotation parameter via application.

       example:

       struct ipu_task       _task;
      _task.output.rotate = 4; (IPU_ROTATE_90_RIGHT) /* for no rotation pass '0' here */
       ioctl( _fd, IPU_CHECK_TASK, & _task )

• And the best thing about following this approach is that no additional CPU usage because everything is taken care by IPU.

For additional information refer commit message available with patch.

By taking reference of my patch on kernel 3.14.28 I have also prepared similar patch on kernel 4.9.11 which latest kernel based on wandbard.

I have attached that patch (framebuffer_rotaion_90_degree_kernel_4.9.patch.zip) here for reference.

Thanks,

Jemish

[Ryze]

I applied the patch in L3.14.52 bsp, but have some issue.

when I poweron the machine,lvds sometimes does not display properly.

kernel error msg:

[ 0.458304] mxc_sdc_fb fb.24: Error updating SDC buf 1 to address=0x3C100000, current buf 1, buf0 ready 1, buf1 ready 1, buf2 ready 0

Can you give me any suggestions ?

[tuban]

I am working with the old Android Jelly Bean release which utilizes linux version 3.0.35. I am adopting a different display to the system, one which is rotated 90 degrees. I hand patched these changes into the mxc_ipuv3_fb.c driver and the system runs as before right up until I 'echo 4 > /sys/class/graphics/fb0/rotate'. At this point I see:

[ 48.135899] imx-ipuv3 imx-ipuv3.0: IPU Warning - IPU_INT_STAT_10 = 0x00080000

And the system hangs.

I am not an IPU guy, so any direction on where to look for this would be great.

[nebilabako]

Hi Jemish,

Thank you for the provided patch("framebuffer_rotaion_90_degree_kernel_4.9.patch") wich could help us also.
we are using YOCTO-2.4(rocko), Vivante GPU driver ("imx-gpu-viv_6.2.2.p0-aarch32") and "XServer-1.19.3".
Using the provided patch to rotate fb with resolution=1280x800, we get a black screen with the following outputs in kernel- and Xorg- log.
Do you think that is due to the restriction of the IPU to a max. resolution of 1024x1024?

Kernel-Log:

###############################################################################################

...
kernel[2176]: [ 4.410249] mxc_sdc_fb fb@0: registered mxc display driver ldb
kernel[2176]: [ 4.416164] mxc_sdc_fb fb@0: Rotate resolution > 1024x1024 is not yet supported
kernel[2176]: [ 4.551092] mxc_sdc_fb fb@0: Error updating SDC buf 0 to address=0x44460000, current buf 0, buf0 ready 0, buf1 ready 1, buf2 ready 0
...
kernel[2176]: [ 39.080355] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_6 = 0x00010000

Xorg-log:

###############################################################################################

...
[ 17.274] (II) VIVANTE(0): printing discovered frame buffer 'fb0' supported modes:
[ 17.332] (II) VIVANTE(0): Modeline "U:800x1280p-56"x0.0 71.10 800 870 880 960 1280 1288 1296 1303 -hsync -vsync -csync (74.1 kHz eP)
[ 17.361] (II) VIVANTE(0): Output DISP3 BG using monitor section LVDS Display Landscape
[ 17.361] (II) VIVANTE(0): Printing probed modes for output DISP3 BG
[ 17.361] (II) VIVANTE(0): Modeline "U:800x1280p-56"x56.8 71.10 800 870 880 960 1280 1288 1296 1303 -hsync -vsync -csync (74.1 kHz eP)
[ 17.361] (II) VIVANTE(0): Output DISP3 BG connected
[ 17.361] (WW) VIVANTE(0): Unable to find initial modes
[ 17.361] (EE) VIVANTE(0): Output DISP3 BG enabled but has no modes
[ 17.361] (EE) VIVANTE(0): xf86InitialConfiguration failed
[ 17.361] (II) VIVANTE(0): VivPreInit: adjust display width 1280
[ 17.361] (EE)
[ 17.362] (EE) Backtrace:
[ 17.362] (EE)
[ 17.362] (EE) Segmentation fault at address 0x18
[ 17.362] (EE)
Fatal server error:
[ 17.362] (EE) Caught signal 11 (Segmentation fault). Server aborting

Thanks!
BR.

[jemish_1990]

Hi Nebila,

Limitation of rotation resolution 1024x1024 is hardware limitation of IC (Image conversion) module of IPU.

Any usecase which includes use IC module (Scaling, Color space conversion, Adjustments: brightness, contrast, color saturation, Combining with a graphics plane) in that case for output frame this limitation will be applicable. It is mentioned in TRM (page: 2736). Thought in TRM they haven't mentioned this limitation for IRT module.

I hope it will clear your doubt.  Let me know if you have any other query.

Thanks,

Jemish

[nebilabako]

Thanks for your quick answer!

as " Jason Haedt" started the discussion with a possible workaround of the limitation of 10214x1024, I wanted to know if this is the cause of the following error output:
kernel [2208]: [4.333392] mxc_sdc_fb fb @ 0: Error updating SDC buf 0 to address = 0x44460000, current buf 0, buf0 ready 0, buf1 ready 1, buf2 ready 0
kernel [2208]: [47.209049] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_6 = 0x00010000

For which resolution was the patch ("framebuffer_rotaion_90_degree_kernel_4.9.patch") successfully tested?
you may have updates to share..
 
Thanks

BR.

[jemish_1990]

Hi Nebila,

One thing in patch I have uploaded print with limitation is warning. It shouldn't put kernel or application to crash.

wanted to know if this is the cause of the following error output:
kernel [2208]: [4.333392] mxc_sdc_fb fb @ 0: Error updating SDC buf 0 to address = 0x44460000, current buf 0, buf0 ready 0, buf1 ready 1, buf2 ready 0

As per my observation this prints are coming few times even in working case, so it is not related to 1024x1024 limitation.

For which resolution was the patch ("framebuffer_rotaion_90_degree_kernel_4.9.patch") successfully tested?

Our requirement was for small resolution it was 240x320 on which it was tested successfully.

For experiment purpose I have verified it on 800x480 also.

Thanks,

Jemish

[nebilabako]

Hi Jemish,

thanks for the given information that help me even if the issue is not solved for me.

BR.

Nebila

[cliffma]

why have the limitation that   "Rotate resolution > 1024x1024 is not yet supported" at function "mxcfb_probe()" according to your patch?

[faustasbagdonas]

Hi Jemish,

first of all thank you very much for your work. I tested this on 4.9.11 kernel with 180 degree rotation but it does not work properly with at least double buffering. We use Qt with eglfs and environment variable FB_MULTI_BUFFER=4 or 2. In case I set LCD refresh rate to 60 Hz and correct LCD timing according its datasheet I don't get any errors related to this driver in kernel log. Without buffering it works well but when I set at least double buffering LCD blinks. It looks like the updated buffer gets exchanged with the old one on the display, but I would have expected the opposite to happen.
I tried to change the LCD refresh rate to 120 Hz and to 30 Hz. In both cases I can notice the aforementioned problem and additionally get the error "Error updating SDC buf". Sometimes the kernel crashes without any message which is really unacceptable for a product.

Could you tell me whether there is a simple solution to fix this problem?

[jemish_1990]

Hi Faustas,

First of all sorry for late reply. I got stuck in other issues.

As per our usecase I have done very basic testing to verify this patch. Actually I am BSP developer so I am using prebuild QT application made by our application team. With that I am displaying GUI (rotated at 90 degree). And in parallel I am rotating video on another layer. This is working very well till now. My current patch still under testing and it will take some more time for any other usecase testing.

Regarding error you facing is there anyway I can reproduce on my setup (Any prebuild qt application)? Because I don't know much about double buffering.  If it is easy to reproduce here please provide me steps so that I can also try here. Without reproducing it I can't say much about it. 

Thanks,

Jemish

[faustasbagdonas]

Hi Jemish,

Thank you for the answer. I tried to fix that myself but without any success. Finally we decided to implement rotation in our applications as a workaround instead of doing that in kernel. After that we migrated from 4.9.11 to mainline kernel and now we use DRM drivers. But even there rotation is not implemented :O .

I think you can reproduce this problem if you use eglfs with Qt and set the following environment variables before application execution:

FB_MULTI_BUFFER=4

QT_QPA_PLATFORM=eglfs

or alternatively add this:

    setenv("FB_MULTI_BUFFER","4",1);
    setenv("QT_QPA_PLATFORM","eglfs",1);

in the beginning of the main() function of your application. Then you should notice that problem if you, for example,  push on a button in your application which opens or closes a window, count seconds or show a slow animation.

Sorry, now I don't have any example application and setup which I had at that time but I hope my mentioned steps will help you to reproduce the problem.

Best Regards,

Faustas

[robertpmclean]

I too am in need of this patch migrated to a 4.1.15 Linux kernel.  Specifically I'm building a Morty release of Yocto and using the meta-freescale layer.

[jingyangxie]

Hi，Robert

      Do  you  find the solution of rotation in linux 4.1.15?if so ,can you share your solution?

[robertpmclean]

If I had a solution or even a reasonable workaround I'd be quite happy to share it.  Unfortunately the best we have come up with is a workaround that only works some of the time, and when it doesn't work we do not have a good explanation. 

The workaround uses Xwindows and "xrandr -o right" to rotate the display.  The the order in which things need to happen is important, and there is a magic delay necessary before calling xrandr which sometimes needs to be 3 seconds but sometimes needs to be 4 seconds. When it doesn't work sometimes when the display comes up blank  but the cursor is moved around, we see what we are calling the "scratch off" effect which allows you to see the refreshed frame buffer but only under the area where the cursor has been.  It's as if the screen refreshes only the area under the cursor, and only when the cursor moves.

[fernando_mercad]

Hi Robert!!!

Did you finally find a solution for the problem?.

I also have the same issue, in my case I use "xrandr -o right" o "xrandr -o left" but with a HDMI monitor, and it's split and the cursor has "scratch off".

I'm using the Kernel 4.9.144.

Can you help me please with some ideas or something.

Thanks.

Regards.