Hello Pat,
If using fbset command, probably it will generate some issues, you can try these operations:
(1) Switching frambuffer: cat /sys/class/graphics/fb2/modes
(2) echo D:1280x800-60 > /sys/class/graphics/fb2/mode
If there still exist erros, It means that your system doesn't support 1280x800 resolution, so you can't get it in your application. You should re-configure display interface( you should confirm which port you are using: LVDS0/LVDS1/ HDMI / Parallel LCD ? ) in Linux device tree( BSP) to be 1280x800.
If you don't know how to port it to your system, please ask BSP or driver softerware engineer to help you do it!
Best Regards,
Weidong