The mxc_v4l2_capture.out app reads frames from the capture device and writes them to a file. It uses the last argument in the command line as the file name and saves 100 frames by default which is why you ended up with a file named "600" that was 72000000 bytes in size. I'm not sure why the larger resolutions would fail though.
Are you trying to stream video to the screen? If so you should look at the mxc_v4l2_overlay.out utility instead.
In case you didn't know, you can view the code for all of the test apps by prepping the imx-test package like this:
ltib -m prep -p imx-test
The files will show up in ltib/rpm/BUILD/imx-test-11.09.01
Judging from responses I've received from Freescale they seem to be moving away from the V4L2 drivers in favor of their IPU library included in the imx-lib package. Have a look at imx5x_IPU_LIB_UG.pdf for more info on that.