A few comments:
For driving an LED you need to be looking at sink current capability, not source, because the LEDs normally have a common anode and the individual segment cathodes are pulled low to turn them on. CMOS I/Os can sink a lot more than they can source.
To save power you can (probably should) look at pulsing power to the LEDs rather than just turning them on. You will lose brightness of course, but this is may be the only way you can ensure the overall power dissipated in the part isnt too high.
We did this with the scrolling display demo on the LPC8N04 board, which actually manages to power the LEDs using only energy harvested from NFC (although the display is pretty dim, and very dependent on the LEDs chosen. That board has 35 LEDs in total, so less than your 4 x 7=21 segments. The I/Os on the LPC84x and the LPC8N04 are very similar.
LPC8N04 Development Board for LPC8N04 Microcontroller (MCU) | NXP
Download the SDK for this board to take a look at the source of the example.
Per segment current can be quite high still, so you have to look at the overall, worse-case consumption (i.e. max segments that will be lit) for your application, to make sure you are not over-stressing the I/Os of the LPC84x device.