MCF5329. It has an internal LCD controller, but you have to add external SDRAM and FLASH. A step up from the "just a simple microcontroller".
On that hardware I can update a complete QVGA screen in 3.6ms with a block memory copy. That's 277FPS.
Maybe you could get an LCD that accepts a serial data stream. The QSPI in the 5329 can send data at 20MHz.
Back of the envelope shows a frame rate of 16 FPS for (320 * 240 * 2 (bytes/pixel) * 9 (bits/byte) / 20MHz) if you're refreshing every pixel, which is worse than you're doing now. If you can just update the bits of the screen that change you should be able to get a faster frame rate.
The MCF52255 has an "external Mini-FlexBus" (whatever that is). That should be able to write data to the panel faster than the GPIO can.