I'll try to make a long story as short as possible (let me know if I skip any important details). Hopefully this is an easy one, but we're still a little green on this part and the IPU is a pretty complex beast, so I'm banging my head at the moment. The LCD we're planning on using for this product is FHD/portrait mode (so, 1080x1920) connected via DSI. We've written a driver for this to glue in to the existing DSI driver (so, pretty much like the hx8369 driver down in drivers/video/mxc). So far so good - everything works at this point. However, the physical orientation of the display will be landscape i.e. 1920x1080, so we were hoping to use the IPU's IRT to rotate the final framebuffer 90 degrees. In order to not have to write plugins for our userspace application(s) to worry about setting up the IPU to do this (and also just to try to get a proof-of-concept going), I've tried to modify the IPU framebuffer driver (mxc_ipuv3_fb.c) to accommodate this. The quick hack to test this out was just to have the driver allocate an extra framebuffer down in mxcfb_map_video_memory (we'll call this new buffer A and the original buffer B). This new buffer is the one that now gets attached to the kernel framebuffer i.e. fb_info->screen_base (so this is the one that will get rendered in to). I set up DMA through the IRT using MEM_ROT_VF_MEM (channel 46 in, channel 49 out) with buffer A being the input and buffer B being the output (initializing both with ipu_init_channel_buffer) (the rotation parameter is set to IPU_ROTATE_90_RIGHT, input resolution is x=1920/y=1080/stride=(3*1920) (we're running rgb/24bpp), output resolution is x=1080/y=1920/stride=(3*1080)). Finally I link the rotation channel to the display channel (ipu_link_channels, MEM_ROT_VF_MEM -> MEM_FG_SYNC).
With this setup, my rotated screen is pretty much garbage (so much so that there's no discernible pattern to guess as to what might be wrong). The weird part though is if I cut down horizontal resolution of the input framebuffer (and of course the framebuffer structure that gets registered) to 1080x1080, the rotation works fine (I believe I've tried non-square resolutions too with a lower x resolution and they worked okay, but I'll have to double-check that one).
Okay, I just did some more digging in to the underlying registers that are getting configured by the IPU driver and I see that the width/height fields in IPUx_IC_IDMAC_3/4 are limited to 10 bits (1024 pixels), so I assume that I'd need to figure a way to break the 1080x1920 rotation in to four separate DMA transfers, correct?