According to datasheet I see that there is limitation on IC output (frame size is up to 1024x1024 pixels). And looks like it was the problem when I have tried to work with 1280x1024 camera - it showed mosaic output but when i've set camera mode to some lesser resolution the image became normal.
But now i'm working with mfw_ipucsc gstreamer plugin and it looks like IC handles heigher resolutions after all. Although this plugin does not handles imagesize direct specification as element option you can use pads in a way like this:
gst-launch videotestsrc ! video/x-raw-yuv,width=320,height=240 ! mfw_ipucsc ! video/x-raw-yuv,width=1280,height=1280 ! queue ! vpuenc ! matroskamux ! filesink location=./test.avi
And this way I receive proper 1280x1280 scaled image. What did I missed? How can it work and to what exactly this 1024x1024 limit applies?