I managed to successfully reinvoke ISP with LPC11U24 using IAP 57 command. I can read the device signature and flash new code using Flash Magic through UART interface. However, when I try to start program execution in Flash Magic (ISP > Execute/Go), UART doesn't seem to work. I can confirm that the program starts executing well, since the LED connected to PIO1_25 lights up (it is turned off right before calling iap_entry function). During the whole process PIO0_1 is HIGH (ISP is reinvoked in application) and PIO0_3 is LOW (UART programming is chosen). If I manually reset the uC, the new code and UART seem to work properly. However, for my applications it is essential that I can enter and exit ISP mode without need for manual reset.
My test code is attached. LPC11U24 returns any character received via UART except for the character 'P'. If 'P' is received, ISP is reinvoked via IAP 57 command.
Any help is appreciated!